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ABSTRACT 


Image understanding for a mobile robotic vehicle is an important and 
complex task for ensuring safe navigation and extended autonomous operations. The goal 
of this work 1s to implement a working vision-based navigation control mechanism within 
a known environment onboard the autonomous mobile vehicle Yamabico-11. Although 
installing a working hardware system was not accomplished, the image processing, model 
description, pattern matching, and positional correction methods have all been 
implemented and tested on a graphics workstation. A novel approach for straight-edge 
feature extraction based upon least squares fitting of edge-related pixels 1s presented and 
unplemented for the image processing task. A simple method for determining the camera's 
location and orientation (pose) follows by matching the vertical extracted edges from an 
image with the linear features of a two-dimensional view of the modelled environment 
based upon an estimated pose of the robot. Image processing, construction of the two- 
dimensional view of the model, and pose determination are conducted sequentially in less 
than one minute for a 646 x 486 pixel image on a 35 MHz processor. The pose 
determination results have been tested to be accurate within a few inches for translational 


error and within one degree rotational error. 
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I. INTRODUCTION 


A. MOTIVATION FOR VISION DEVELOPMENT 


Image understanding has been a difficult subject in the fields of artificial intelligence and robotics for 
nearly twenty years. The prospect of developing a novel and revolutionary approach to image processing and 
image understanding in the course of writing a master’s thesis seems unlikely. Instead, as with most all vision 
systems developed, this approach will be directed towards developing a visual navigation system based upon 
the specific needs of the robot and tailoring the application to the domain of its operating environment. 
Certain assumptions will be made to simplify the problem. These assumptions are: 


- the robot will operate in an indoor, orthogonal environment, 
- the floor will be a flat surface, 

- the floorplan of the environment will be known, 

- the robot will always be in an upright position, 

- and approximate position information will be accessible. 


The image understanding techniques to be described have been developed utilizing a charge-coupled 
device (CCD) television camera, a graphics workstation, and a video frame grabber. The CCD television 
camera (JVC model TK870U) is capable of providing digitized RGB formatted color information (red, green, 
and blue components) to an image processing system. The graphics workstation is a SiliconGraphics Personal 
Iris and the video frame grabbing board is SiliconGraphics’ VideoFramer. The SiliconGraphics graphics 
library supports basic display functions in the C language. The vision based routines described within this 


thesis are programmed in the ANSI C language. 


B. AUTONOMOUS MOBILE ROBOT SYSTEM YAMABICO-11 


Yamabico-ll (Figure 1.1) is an autonomous mobile robot that has been the focus of many students’ 
theses at the Naval Postgraduate School. Work in the fields of motion control, safe path planning, and sonar- 
based navigation are continually ongoing to upgrade the robot’s capabilities. Yamabico’s platform is an 
aluminum cart that stands 36 inches tall, with two main wheels for motion and steering and four smaller 
wheels for support. Its maneuvering capabilities resemble that of a tracked vehicle (it can pivot in place). 
Power is provided from two 12-volt wheelchair batteries. 

Process control is conducted by one Motorola 68020 32-bit microprocessor with an accompanying 
Motorola 68881 floating point coprocessor. The processor has 1Mbyte of onboard RAM and runs at 16MHz. 


Control information is passed over a VME bus which also carries a dual axis controller board and two 4- 


channel serial communication boards. The dual axis controller interfaces with dnving and braking motors. 
The serial channels interface with an onboard terminal and a modem to a host computer. 

Its only sensors, twelve ultrasonic sonars that operate at 40kHz, are mounted about the lower base and 
are controlled by three Motorola 6809-based microprocessors. The Sonar Control Language [SHE91] and the 
Mobile Motion Language (MML) [KAN89a] provide easy sonar and motion control features to the 
programmer and are downloaded to Yamabico via an RS232 serial port from the host computer, a Sun 
Microsystems 3/160. The system is composed of a kernel and a user-program. The kernel (about 100 kbytes) 
only needs to be downloaded once. The user-program, user(), specifies Yamabico’s tasks. Once the object 
code has been downloaded and the serial link disconnected, all computation is performed autonomously and 
is executed via a keyboard mounted on top of Yamabico. 

Presently, Yamabico can navigate its way through a “known” corridor by utilizing ultra-sonic range 
information to update its position which is maintained by wheel motion dead-reckoning. Yamabico’s ultra- 
sonic sensors are effective in ranges from 9.6cm to 400cm and are accurate to 18.95mm at a range of 500mm. 
The range information can directly update the current position in a known environment (such as our testing 
grounds, the fifth floor of Spanagel Hall at the Naval Postgraduate School). Prior to the work outlined in this 
thesis, Yamabico had no image based sensor capabilities at all. The purpose of this thesis is to develop and 
implement a vision based sensor system for Yamabico-11 suitable for navigation and exploration of its 


environment. 





Figure 1.1 The autonomous mobile robot Yamabico-/ 1. 


mee LITERATURE SURVEY 

Image understanding has been a complex topic for computer scientists for almost two decades. The 
ideas and algorithms uulized in this work stem from the image processing, linear fitting, pattern matching, 
and pose determination methods that have been found to be successful in past applications. Two good sources 
for the basic ideas necessary in image understanding are by Ballard [BAL82] and Nevatia [NEV82]. These 
are both textbooks that outline the more commonly used methods and the background fundamentals. They 
cover a wide range of topics including low level image processing, graph-theory for pattern matching, and 
model-base object recognition. 

A paper by Marr | MAR79] outlines requirements for model-based image understanding and provides 
insight for construction of suitable models, similar to his “2D+,” model for vision applicauons. Stein’s thesis 
[STE92] describes the environment model devcloped for the vision navigation method developed in this 
thesis. Two papers by Kanayama, [KAN9Oa] and [KAN9O0b], were consulted for familiarization with the 
robot Yamabico’s mouon control capabilities (with the MML language) and the application of linear fitting 
ol sonar point data by least squares for environment feature mapping. Sherfey’s thesis [SHE91] provides the 
dciails on Yamabico’s sonar capahiliues. 

A lotot recently published research conducted at the University of Massachusctts at Amherst provided 


motivauon to develop faster und simpler image edge extracuion and visual navigation rouunes. Papers from 





this group by Beveridge {[BEV9O)|, Kumar [KUMS89], and Fennema [FEN90] describe methods for pattern 
matching, pose determination, und their experimental results. Edge extracuion was conducted via the Hough 
transform and pattern matching performed by random combinauons of image to model line matches. 

Pose determination methods described by Haralick [HARS89] and Liu [LITU9Q] are algonthms based 
upon the general uigonometric principles of photogranimetry. A photogrammetry textbook, such as Moffit’s 
[MOF80], proved to be a good source to explain these fundamentals. During development of this algorithm, 
a means lor pose verification was found to be necessary. Methods tested by Heller [HEL89] show the results 
of verifying all image Imes with model lines and the results ol verifying model lines with edge-related pixel 
data. A basic computer graphics teatbook by Hall [HAL89]} was also consulted for checking video source 


formats and computer Image data formats. 
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D. THESIS ORGANIZATION 

The following Chapters outline the development of the algorithm applied towards a simple visual 
navigation scheme for an autonomous mobile robot in a known orthogonal environment. The objectives are 
stated in Chapter IT. Chapter II] presents the general method of feature extraction based upon determining the 
edges depicted in an image. Chapter IV discusses the method in which the image’s pixels are grouped into 
two-dimensional regions describing the edges. Extraction of straight line segments from two-dimensional 
areas based upon least-squares fitting of datapoints is presented in Chapter V. The straight edge extraction 
algorithm is then outlined in Chapter VI. The algorithm’s results and the effects of varying parameters within 
the algorithm are shown in Chapter VII. Chapter VIII discusses implementation factors such as image data 
structures and the environment model. 

Chapter IX discusses the method for pattern matching of extracted edges from the image with modelled 
environment features. Chapter X describes pose (the vehicle’s position and orientation within the 
environment) determination based upon disparities between the image and the expected view of the modelled 
environment from an assumied position. Pose determination results are shown in Chapter XI. Conclusions and 
subsequent topics for this visual navigation system are presented in Chapter XII. 

Appendix A specifies the atan2 function used for determination of a vector orientation and it contains 
the data types header files used. Appendix B contains the edge extraction implementations findedge and 
fastedge and all the required header files for compilation. Appendix C contains the implementation vermnatch 
which extracts vertical image lines, matches them with the model lines from an expected two-dimensional 
view of the environment, and determines the robot’s correct position and orientation. Appendix D contains a 


user's guide for the compilation and operation of the edge extraction and pose determination programs. 


Hl. OBJECTIVES 


There are two primary objectives for the development of a vision understanding system for the 
Yamabico-!I1 robot. They are similar to the objectives for any autonomous mobile vehicle. 


1. Update correct vehicle position and orientation for precise navigation. 
2. Recognize specific objects using u prior knowledge based upon the task requirements and the 
operational environment. 


A. POSITION / ORIENTATION IDENTIFICATION IN A KNOWN WORLD 


Determining accurate position and orientation autonomously is a crucial requirement for the successful 
and safe navigation of any mobile robot. This task is also known as pose determination and is used to update 
the robot’s positional information that is maintained by wheel motion dead-reckoning. Small errors from 
dead-reckoning arise quickly and are compounded by turns and acceleration. Continuous periodic pose 
determination is fundamental for precise dead-reckoning based navigation. Chapter X outlines a simple 
position-correction algorithm via pattern-matching of the extracted features from a two-dimensional video 
image and two-dimensional view of mapped features of the three-dimensional environment model 
maintained by the robot. This algorithm is based upon previous works [HAR89], [MAR79], [MOF80], 
[KUM89}], [FEN90], and [LIU90], whose roots stem from shipboard navigation and aerial photogrammetry. 


B. OBJECT RECOGNITION IN AN UNKNOWN OR PARTIALLY KNOWN WORLD 

The second objective is the recognition of specific objects via three-dimensional models. This is a 
much more complex task than the first due to the uncertainty of individual objects presence and location. The 
first task of pose determination is a subsct of this task. The operating environment is modelled as a three- 
dimensional object and determining the camera's pose is only determination of viewing aspect of the object. 
Object recognition covers many other topics including mapping an unknown environment, searching for a 
specific goal, classifying encountered objects, and avoiding unknown obstacles. Some aspects of object 
recognition are detailed in [BAL82], [MAR79}], [NEV82], and many others. A central theme of all vision- 
based object recognition applications is modelling and searching for only the objects that are expected to be 
encountered and classifying unknown items by their geometrical properties. This objective is not approached 


within this paper, but will hopefully be accomplished for Yamabico by a successor. 


I. METHOD 


A. ARCHITECTURE FOR IMAGE BASED NAVIGATION 


Extracting desired features from an image is a crucial process in any image understanding 
implementation. Some methods focus on determining areas of common light intensity, color, or texture. 
These methods often include segmentation of the image by region growing and are generally suitable for 
images with small areas of pixels with common attributes (e.g. a picture of a forest). Other methods search 
for the edges between contrasting bordering areas of pixels with similar attributes. Likewise, these methods 
are suitable for images with large areas of common light intensity, color, or texture (e.g. a picture that 
included bare walls). This method will be of the latter category; extracting the edge features of an image. This 
method was chosen for recognizing the straight edges of orthogonal objects whose surfaces are often uniform 
in color and texture. The images that the edge extraction methods will be applied to, will be pictures of the 
interiors of offices and hallways. Pictures where straight line segments would be common and suitable for 
describing the objects in the image. 

The goal of the image processing part of the vision system will be to find straight line segments in the 
edges of boundaries between areas of similar light intensity in an image. The straight line segments should 
be a sufficiently simple data structure for use in follow-on image understanding implementations. The 
specific problem of image understanding for navigation of Yamabico in a known, orthogonal environment 
will be split into the following subproblems: 


1. Image Processing - extracting desired features (straight line edges) from the input image. 

2. Pattern Matching - correlating extracted features with known features described in a three- 
dimensional model of the environment. 

3. Pose Determination - calculating the position and orientation of the robot within its modelled 
operating environment. 


Since Yamabico has no vision based sensor capabilities, the following features were identified for the 
implementation of a vision understanding system (Figure 3.1) 


- a three-dimensional model representation of the operating environment, 
- camera image processing functions, 

- image feature extracting functions, 

- pattem matching routines, 

- and visual-based position correction methods. 
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Figure 3.1. The general outline for image understanding on Yamabico-11. 


In order to accomplish the image processing task of finding straight line segments in an image, the 
borders (edges) between areas will be identified, segmented, and then simplified in terms of straight line 
segments (Figure 3.2). The identification of the edges in an image (Figure 3.3) 1s best described by a gradient 
image (Figure 3.4). Two important steps to follow are: 


1. grouping of pixels into contiguous regions of separate edges, and 
2. linear fitting of the edges. 


GRAYSCALE IMAGE 


Extracting Edge Regions 





EDGES 


Application of Least Squares Fit 
method on Edges 





LINE SEGMENTS 





Figure 3.2. Outline of method for edge extraction of an image. Edges will be represented 
as line segments. 


Thus, once the gradient image is made, pixels that define the edges (the black pixels of Figure 3.4) must 
be grouped and then fitted to a straight line segment by least squares fit to the major axis of the edge region. 
Determining when to start and stop constructing separate edges is a crucial subject. Checking the validity of 


the line segments will be the final requirement of the image processing task. 
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Figure 3.4 The gradient image for the chair in Figure 3.3. Edges between regions of 
common light intensities are depicted by black pixels. 
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Figure 3.5. The extracted line segments based upon the grouping of similar edge-rclated pixels. 


The pattern matching process will require two inputs. These will be the linear features from the three- 
dimensional model mapped onto a two-dimensional viewing plane and edges extraeted from the image 
processing system. The three-dimensional model is a two-dimensional floorplan plus vertical eomponents for 
height measurement. This model will be used to construct a two-dimensional “wire-frame” view of the 
environment from any given pose for the robot’s camera. This view will be used for what the robot will expeet 
to sce from its present locauon and orientation. The three-dimensional modelling system for Yamabieo 1s 
being pursued concurrently with this work by Lt. Jim Stein for his master’s thesis [STE92]. 

The image processing system could be executed in parallel with the three-dimensional modelling 
System as it provides an updated two-dimensional wire-frame view of the robot’s orthogonal environment. 
The pattern matching process will correlate the extracted edges from the image with the known features from 
the model. 

The task of pose determinauion will follow the matehing of the three most significant edges with all 
possible vertical model lines. The observed differenee of horizontal viewing angles between the vertical 
image edges will be fitted with the map loeations of vertical model lines to determine the only possible 
position and orientation of the robot’s eamera. Many different possible values for the robot’s pose will be 


ealculated as different line matchings are tested. 


A method for pose verification will be necessary to select the best possible pose (hence, also the best 
matching) to be retummed as the correct pose of the robot. The correct pose can then be used to update the pose 
maintained by dead-reckoning. Sensor fusion with the present sonar system could possibly follow for 


confirming the accuracy of the visual fix for the pose. 


ie 


IV. GENERATING REGIONS OF EDGES 


A. GRADIENT IMAGE 


The first requirement for the image edge extraction algorithm is the determination of high contrast 
regions between the areas of similar light intensity within the image. These regions of significant light 
intensity contrast will define the edges. The CCD camera simplifies the quantification of light intensity by 
discretization of the image into separate pixels. Chapter VII identifies some of the formats provided by the 
camera and used in the graphics workstation. The pixel format utilized herein is the RGB format which 
specifies the red, green, and blue color attributes of the pixel and is described fully in Chapter VII. Differences 
between the areas of the image are therefore determined by differences in pixel intensities. The simplest way 
to consider pixel intensities is in terms of grayscale values vice considering the separate associated color 
attributes of each pixel. The color to grayscale conversion of RGB pixels in also described in chapter VI]. 
Determining the differences in grayscale intensities between adjacent pixels in a two-dimensional image will 
result in a gradient value for each pixel. The pixel gradient is a vector whose magnitude represents the amount 
of light intensity change between adjacent pixels, and whose angular orientation is directed towards the 
lighter pixel. Pixel gradient values will be determined in a two-dimensional cartesian coordinate plane by the 


amount of change of grayscale intensities in both the horizontal (dv) and vertical (dy) directions of the image. 


1. Gradient Window Operators 


A common method to determine pixel gradients is by the use of gradient window operators. Two 
gradient window operators, one for determining the change of pixel intensities in the horizontal (dx) direction 
and another for the vertical (dy) direction, must be specified. Gradient window operators are square matrices 
of weights, mapped onto a group of pixels about a central pixel (or point). The weights are multiplied with 
the intensities of the surrounding pixels and then summed to provide values for the intensity changes in the 
horizontal and vertical axes. Commonly used operators are the Four-Square, Prewitt, Sobel, and Schirai 


(Figures 4.1 through 4.4). 
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Figure 4.3. The Sobel gradient windows. 
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Figure 4.4. The Schirai gradient windows. 


To be geometrically correct for a two-dimensional plane of pixels evenly spaced in both honzontal 


and vertical directions, a modified Sobel operator should be used with values of ./2 vice 2 for weights of the 
non-diagonal pixels (Figure 4.5). This would assume that all of the pixels sensors on the CCD array in the 
camera are in a perfectly square grid. However. since one of the assumptions of this vision system Is to operate 
in an orthogonal environment, the standard Sobel operator could prove to be more valuable since it enhances 
the vertical and horizontal edges found in the orthogonal environment. After implementing and testing all of 
these window operators, the Sobel seemed to provide the strongest edges and the least amount of smaller, 


insignificant and undesirable edges . 





Figure 4.5. The modified Sobel gradient windows. 
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Once the values for dv and dy are calculated by the gradient window operators, the gradient is 
easily calculated. Pixel gradient magnitude (|V (pixc/;)| ) and gradient angle (V «(pixel;) ) would then be 
determined by equations 4.1 and 4.2. The atan2(dy,dx) function is common in many programming languages’ 


standard math libranes and 1s described in Appendix A. 


IV (pixel,)| = (dx) + (dy,)” (4.1) 


Va(pixel,) = atan2 (dy,, dx.) (4.2) 


To determine if a pixel is part of an edge between two areas of an image, the gradient magnitude 
must be greater than a specified threshold value, C;. Such a threshold can be determined dynamically by 
scanning through the image once to determine the average weight of pixel intensities or by maintaining a 
histogram of previous images’ average pixel intensities. An alternative approach is to keep the gradient 
magnitude threshold value as static value, determined by testing. This method would be suitable for a robot 
operating in an environment with relatively constant illumination and alleviates the requirement of an extra 
scan through the image to dynamically determine a value for C;. This method will be pursued based upon the 
assumption that the robot will initially be tested in an indoor environment with relatively uniform lighting, 
whereas the other method for threshold calculation would be more suitable for the vision system adjusting to 


major environment lighting changes (e.g. the robot travelling into an unlit room). 


2. Construction of the Gradient Image with the Sobel Operator 


The Sobel gradient window operators will be utilized to calculate pixel gradients from the 
grayscale values for each pixel. The Sobel operator is defined as a two-dimensional matrix; that is to be 
implemented upon pixels read and stored in a one-dimensional array (see Chapter VIII for description of the 
data structure for the image). Using the image’s horizontal dimension (xsize) as an offset for finding the 
adjacent pixels in the neighboring rows of a two-dimensional image. equations for calculating the dx and dy 
values for the gradient of a central pixel, 7. are defined by equations 4.3 and 4.4. A pointer (ptr). is utilized to 
access each grayscale pixel value from the one-dimensional array in the data structure for the image. 

oe sise 1 | ptt size + 1] 
= —2ptr[i—-—1) +2ptr[i+1) (4.3) 
= pti [i= vsize — 1) + ptr |i —\VSsiee tal) 


lhe 


F ptr [itxsize —1] + 2ptr [it+xsize] + ptr [it+xsize +1] (4.4) 
y= 
| = ptr [t= xsize V2 pir |i Xs ze Va ale eae 


Notice that in order to calculate the Sobel operator for pixel i, pixel i+xsize+J/ and all previous 
pixels must have a grayscale values already determined. A possible method could be to first construct a 
corresponding grayscale image of the input (color) image followed by construction of the gradient image 
using the Sobel operator equations. Pixel gradient magnitudes larger than the predefined threshold (C;) would 
then be set to be black in the gradient image and all pixels would be white. This would provide a gradient 
image such as displayed in Figure 3.4. A simple algorithm to do this would be: 


- Read in input image. 

- Allocate memory for new grayscale and gradient images. 

- Build the grayscale image by grayscale conversion for all pixels. 

- Build the gradient image via Sobel operator for all pixels. 

- If pixel gradient magnitude is greater than the magnitude threshold, C,,then set the 


corresponding pixel in the gradient image black, else set the corresponding pixel white. 


3. When Not to Use the Sobel Operator 
However, not every pixel in the input image (more precisely, the grayscale image) can have their 
associated gradient values properly calculated. The outermost pixels, those in the top and bottom rows and 
the leftmost and nghtmost columns will not have gradient values since the Sobel operator requires eight 
adjacent pixels about the central pixel gradient value. Calculating gradient for these outermost pixels will only 
produce errors when using the gradients. Therefore, the dimensions of the gradient image are two pixels less 
in both horizontal and vertical directions and careful consideration must be paid towards scanning through 


the input image in order not to calculate pixel gradients for the outermost pixels. 


B. FINDING CONTIGUOUS REGIONS OF PIXELS TO DESCRIBE EDGES 

Since the black pixels in the gradient image represent pixels with gradient magnitudes sufficiently 
large to be considered part of an edge, each black pixel must be considered for grouping into contiguous 
regions that define the edges. Therefore, grouping pixels into regions will be performed upon every pixel with 
a gradient magnitude larger than the threshold. C;. Determining a suitable value for C, is critical to the 
success Of this algorithm. A threshold that is too high will not allow for the detection of all significant edges 


with those that are found being broken and incomplete. On the other hand, a threshold that is too low lead to 
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edge finding and linear fitting of non-edge areas of the mage. Chapter VII shows results of varying the 
gradient magnitude threshold C). 

After a suitable threshold is determined and used, the problem of grouping edge-related pixels is to 
define how two pixels share features common to an edge. Figure 4.6 can be considered as a portion of a 
gradient image with two distinct areas of common light intensity (areas A and B) and the subsequent edge in 
between, compnised of pixels with gradient magnitudes greater than the given threshold. In order to describe 
the edge, each pixel must be grouped into one contiguous region. There are three important aspects of a pixel 
that can be used for this comparison: pixel gradient magnitude, pixel gradient angle, and pixel adjacency. 


Pixel gradient magnitude has already been considered by testing with threshold C). 
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Figure 4.6. A region of pixels defining an edge in the gradient image. A and B are areas of 
common light intensity (pixels that are white in the gradient image). Area B is darker than 
A, subsequently all edge pixel gradients will be directed towards A. Similarly all edge pixel 
gradient angles will be close to the normal of the line segment describing the edge. 


For two pixels to be of the same edge. they must both have gradient angles that are close. “Closeness”’ 
can be defined by ensuring the difference between two angles is less than some constant angle, C >. The pixel 
gradient orientation ts the normal to the tangent of the border between the light and dark areas of the image 
at that point (areas A and B in Figure 4.6). Therefore. all pixel gradient orientations wull be close to the normal 
of the line segment describing the edge region. If a pixel’s gradient angle is not close, then the pixel can not 


be considered to be part of that edge. The effects of varying C> are shown in Chapter VI. 


17 


The adjacency requirement can readily be seen in Figure 4.6. For a contiguous region of pixels to 
define an edge, all pixels in the edge must be adjacent to at least one other pixel in the edge. Thus, after testing 
a pixel’s gradient magnitude for description of the gradient image, only the pixel’s gradient angle and the 
pixel’s adjacency with a previously edge-defined pixel needs to be considered (Figure 4.7). If these 
requirements fail, the pixel must still be considered being a member of an edge and therefore it will be the 


first pixel to define a new edge. 


Gradient Image 





direction of scan 
row presently belng scanned 


4 pixels to check 


Figure 4.7. Pixel / is only compared with four adjacent pixels to check gradient orientation 
closeness. One of these four pixels must already belong to a defined edge. 


C. DECIDING WHEN TO STOP ADDING PIXELS TO AN EDGE 

Continually declaring new edges for every pixel that does not meet the membership requirements for 
a declared edge will rapidly decrease performance as the number of edges gets large. It is desirable to process 
edges for linear fitting once it has been determined that no more pixels could possibly be included. This ts 
done by enforcing the eight-pixel adjacency rule for pixel inclusion in an edge. Every time that the scan of 
pixels starts at a pew row in the image, all edges must be inspected to ensure that the last pixel added to each 
was from the row just scanned (Figure 4.8). If this check fails, then no pixels found on the present row (the 
row to be scanned) could be included with that edge, therefore that edge can be considered completed for 
grouping of contiguous pixels and may be processed for linear fitting. Thus all edges that pass this test are 


active edges that can still accept more pixels for inclusion. 


18 


Gradient Image 














ready to start last pixels added to edges 


scanning next row 





row just scanned 


Figure 4.8. Prior to starting the scan of a new row, the list of edge regions must be reviewed to ensure 
that the Jast pixel added to each edge occurred in the row just scanned. If not, pixels found on the new 
row could not possibly be associated with the edge. Thus, edge A would be removed from the edge 
list whereas age B is retained as an “active” edge that could still accept more pixels for inclusion. 
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V. LINE FITTING BY LEAST SQUARES 


A. LEAST SQUARES FIT METHOD 

One method for determine a line segment that adequately describes a two-dimensional region of 
datapoints is by least squares. Linear fitting by least squares is a simple and efficient method for calculating 
thousands of datapoints. In an image, the least squares fit moments, described by Kanayama [KAN90b], of 
pixel locations (x,y pixel coordinates of the image) for all pixels in the edge region will yield an associated 
line segment for the region’s major axis. Verification of the region is required to ensure that the least squares 
fit moments are representative of a line segment. Regions that meet the requirements for a line segment are 
saved for the desired output and all others are discarded. 

Once the decision has been made to include a pixel as a member of an edge region, the least squares 
fit moments must be updated for the region to include the new pixel. Consider the image as a cartesian 
coordinate system, discretely numbered by pixels in both honzontal and vertical directions. Each edge region, 
R, can be considered as a collection of pixels occupying a two-dimensional space (Figure 4.6). Pixels’ 


positions (numbered by pixels in x and y axes) are used to define the least square fit moments 7. 
1. Primary Least Squares Fit Moments 


Each moment mj, (O SJ, A S 2 and j + k S 2) of an edge region, R. is defined as: 


rt 
Ph deak 


(| 
Considenng that gg is equal to 1, mgg defines the number of pixels associated with region, R. 


The centroid C of an edge region, R, is defined as 
C= =" — |= (Me) (5.2) 
O00 


2. Secondary Least Squares Fit Moments 


The secondary moments about the centroid are: 


2 
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3. Parametric Line Fitting of the Principle Axis 
Using a parametric representation (r,a) of a line L, a point at coordinates (x,y) satisfies the 
equation 


1G TU 


i= eveas © + y sine ry (5.6) 


The residual of pixel i (located at (v,. ¥,)) and the line L, defined by 4,, is the distance perpendicular 
to L and pixel 1. 
Oe) COsO caging) sinc, (5.7) 
The projection of pixel t, Dis onto the line L is 
Oe COSC a0 SI) (5.8) 
Therefore, p’, and p’_, the projections of the first and last pixels associated with the edge region R onto the 


line segment L, can be used as estimates of the endpoints of L. 


The sum of all the residuals is then 


S = BS (1 — x,cosa — y,cosa) *. (5.9) 


i= 
The line that best fits the set of points will minimize the sum S. Thus, the optimum line L (7,a) must satusfy 


dS _ dS 
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Therefore, 


rn 


3 2 BS (7 —X,cOSO — y. Sing) 


dr ; 
p=) 


- 2 (-3.1}- (Sema (Fx }uinl 


f= t=) 


= 2 (rig — 1, 9COSA — Mp; SIN) 


mae (5.11) 
and 7 can be expressed as 
0 LES ie 
r= —cosa+ — sing® = UW cosa+p sing, (5.12) 
m m - y 
00 OO 
where r may be negative. Substituting r 1 equation 5.9 by equation 5.12 
rt 

> (C4; - H,) cosa + (y;- HL,) sino) a (5.13) 

r=1 
Finally, dS/da can be solved for. 
aS -> ((4,——.) cosa (7, —H, J SHO) ea ) sing + ()- 1) cose) 
da y 

} — — —_ 2 ' 
x ((¥)—B,)*- (4-H) *) sinacosa 
a rn 
: 2 2 
+ >) G;- HD (Y;- By) (cosa) “- (sina) 7) 
i=] 
I 
(5.14) 
Which provides a solution to the normal of the line L, a. 
a= satan [ cae (5,15) 
2 Mo. = Mo 


Equations 5.12 and 5.15 are the solutions for the line parameters generated by a least squares fit 
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B. TESTS FOR VALIDITY OF AN EDGE REPRESENTING A LINE SEGMENT 


The equivalent ellipse of inertia (Figure 5.1) for the edge region, R, will have the same moments about 


the centroid (A499, Mf) 1, and Moz) as R. Minajor ANd Myningy, the moments about the major and minor axes of 


the ellipse, are defined as: 


2 
(Vein Cif. —M.-) 
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The lengths of the major and minor axes, dygior ANd Amino, are defined to be: 
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Figure 5.1. The major and minor axes of the equivalent ellipse of inertia for the 


region of pixels defining an edge. 
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For this implementation of least squares fit, the major axis will be considered to be the associated line 


segment for the edge region. The endpoints of the major axis will be the endpoints for the desired line 
segment. The projections of the first and last pixels, p” j and ee associated with the region R, will therefore 


be used as estimates for the two endpoints of L. A ratio, p, of the axes length can then be used to describe the 


thinness of the ellipse. 
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Using the values for number of pixels (rg), major axis length (d,,gj9,), and ellipse thinness (p) as 
parameters for comparison to edge significance, edge length and how much the edge resembles a line can be 


simply tested as long as the least squares fits moments are maintained for every pixel included in an edge. Let 
C; (OSC,S 1) be aconstant for the maximum allowable ratio p that can be used to described a line. Let 


Cy be a specified constant for the minimum number of pixels and Cs; for the minimum line length. Three 


requirements can therefore be specified for the line testing of an edge region. 
]. the ratio of axes lengths less than the maximum ratio (p < C3), 
2. the number of pixels greater than the specified minimum (rp9 > C4), and 
3. the line length of the region greater than the minimum length (d,,4;, > Cs). 
The ratio P proves to be the most significant measurement. For p to equal 1.0 means that the length of 


the minor axis is equal to the length of the major axis, representing an edge region that resembles a circular 
blob. Therefore, p can be compared to a maximum ratio, C3, specified by the user. C; equal to 1.0 allows all 
edge regions to be considered thin enough to represent a line segment. A value of 0.1 seems to work well. 

The two other tests can be used to trim down the number of smaller, less significant edges found. 
Specifying minimum number of pixels in an edge and minimum edge length, Cz and C, respectively, is an 
effective means to filter out less significant line segments. The regions that meet these requirements will be 
saved for the desired output as the found line segments and all other regions will be discarded. The effects of 
varying the parameters C3, Cz, and Cs are described in Chapter VI. 
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VI. ALGORITHM FOR EDGE EXTRACTION 


From the principles stated in the previous chapters, the following elements were identified to be 
incorporated the edge extraction algorithm: 


I. Determine which pixels compose the edges. 

2. Group pixels together into regions describing individual edges. 

3. Describe a completed edge region in terms of a line segment via least squares fit of pixels’ 
positions. 

4, Ensure that line segment representation of the edge meets the specified thinness and length 
requirements. 


The implementations are coded in the ANSI version of the C language and are contained in Appendix B. A 


user’s guide for compiling and executing the implementations is in Appendix D. 


A. THE FINDEDGE METHOD 

Findedge is a C-program that follows the above outline. Remembering that the grayscale value for 
pixel i+xsize+/ must be determined prior to the gradient calculation for pixel 7, the algorithm for findedge 
can be stated to build both the grayscale and gradient images in one pass of scanning the input (color) image. 
This algorithm will ensure that gradient values are not calculated for the outermost pixels. Pixels with 
gradient magnitudes greater than the specified threshold, C,, will be considered as datapoints for the edges. 

Two tests were identified for a an edge-related pixel to be included as a member of a particular edge. 
Those tests were: | 


1. The pixel must be adjacent to a pixel already included in an edge. 
2. The pixel’s gradient angle must be “close” to that of the averaged angle of adjacent pixel’s. 


Two angles are “close” if the difference between them is less than a maximum constant, C,. These tests 
can be conducted immediately once a pixel has been identified to be an edge-related pixel. Therefore, all 
black pixels in the gradient image will be tested for edge membership. 

Pixels meeting the edge membership requirements must then be incorporated with the edge by 
updating the least squares fit moments, mx. for the edge region, R. This is accomplished by determining the 
ith pixel’s two-dimensional image position (x,¥) and summing the moments, mm,,. with the corresponding 
weights. If a pixel fails the membership test for the declared edges, then that pixel will be used as the first to 
declare a new edge. Thus, a linked list is the preferable data structure to maintain the declared edge regions. 

Observing that new edges are declared as a linear scan of the image is conducted. and that the first 


requirement for pixel membership with an edge is adjacency, a pixel needs only to conduct the membership 
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test with the pixel to the immediate left (assuming a left-to-right, bottom-to-top scan). This builds edge 
regions for the present row only. Pnor to continuing the scan at the start of the next row, the list of edge 
regions for the row just scanned will be compared with the list of edge regions from the previous row to see 
if they can be combined based upon vertical adjacency of edge regions between the two rows and “closeness” 
of average orientation of pixels’ gradient angles. 

Maintaining values for the first and last pixels included for each edge is necessary for the determination 
of line segment endpoints. Keeping track of first and last pixels added is straightforward with only one 
exception for horizontally oriented lines. The case for horizontal lines (Figure 6.1) requires determination of 


endpoints based upon the leftmost and nghtmost pixels of the edge vice the first and last pixels added. 


General Case of Determining Line Segment Endpoints 
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Figure 6.1. Genera] and horizontal cases for determintng pixels to be used for calculation of 
line segment endpoints. The first and last pixels in the horizontal case may not necessarily 
fully describe the extent of the associated line segment. 1n which case the leftmost and nght- 
most pixels should be used. 
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Edges from the previous row that are not appended with pixels from the row just scanned are 
considered finished with their construction and can there fore be tested for meeting the three requirements for 
a line segment. These requirements were identified to be: 


1. The ratio of lengths of major and minor axes (p) < C;. 
2. Number of pixels in an edge (nya) > C,. 
3. The length of the major axis (Amajor) > Cs. 


Determining dyagjor and p is easily done since the primary least square fit moments are updated for 


every pixel added. Edges that meet these requirements will serve as the desired output of straight edges, 


otherwise they are discarded and the associated memory is freed. 


B. AN ATTEMPT TO CUT COMPUTATION COSTS 


The findedge algorithm requires an averaged value of the pixel gradient angle orientations to be 
maintained for each edge. This average value is used for the comparison with pixel gradient orientation to 
determine angle “closeness.” Thus, for every pixel added to an edge, the average of the pixels’ gradient 
orientation must be calculated. To alleviate these computations, the specified maximum value for angle 
closeness, C,, could be used to determine two Static sets of angles that any pixel gradient orientation could be 
defined by. These two sets could be thought of as dividing up all possible angles between x and -z by C, as 
illustrated in Figure 6.2. 

“Closeness’ could now be defined by comparing the two groups that a pixel gradient angle belongs to 
with the groups of the adjacent edge-related pixel’s gradient angle. If either groups are similar, then the 
gradient angles are close. This method is termed static since the two sets of angles are computed only once 
and remain unchanged whereas the findedge method continually recomputed the average gradient angle of 
the pixels in the edge region. The results of this method of pixel grouping into straight line edges are shown 
in Chapter VII. Compared with the line segments from the findedge algonthm, more line segments are 
extracted but shorter and more fragmented. Since the desired output is a simple structure (line segments) that 


best represents the linear features in an image, the performance of the static method is worse. 
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Division of angles by Co = 20.0 degrees 
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Figure 6.2. The two static groups of angles determined for C, = 20.0 degrees. Each pixel 
gradient orientation will belong to one region from each group of angles. The groups can 
be thought of as overlapping in such a way so that “close” gradient angles not belonging 
to the same region in one group will belong to the same region in the other group. 


C. IGNORING GRAYSCALE CONVERSION WITH THE FASTEDGE METHOD 


Computation costs can still be reduced in the findedzge algorithm by omitting the construction of the 
grayscale and gradient images. Observing that the green component of the RGB color format accounts for the 
majority of grayscale weights, pixel gradients can be estimated by using only the green component value. This 
is accomplished with a bitmask to get only the green color values for use with the Sobel operator (equations 
4.3 and 4.4). Thus, construction of the grayscale image is unnecessary. Similarly, by observing that the 
gradient image only identifies those pixels that describe edges, and otherwise not required for the construction 
of the edges, its construction and storage as a real image will be omitted. 

Therefore, the program fastedge can perform the same straight line edge extraction as findedge; 
however, since the values for pixel gradient calculation are only the green color components and not the 
weighted grayscale sum, a new gradient magnitude threshold value (C,) must be determined. The fastedge 
implementation displays the input (color) image and draws the extracted lines over the image. Results are in 


Chapter VII. A guide for compiling and using fustedge is contained in Appendix D. 
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VIL. IMPLEMENTATION 


A. SYSTEM DESCRIPTION 


The graphics workstation used for feature extraction is a SiliconGraphics Personal Ins. Its graphics 
support libraries are implemented in the C language: therefore, the code for the findedge and fastedge methods 
is in C. The VideoFramer frame grabbing hardware is a daughterboard on the Iris that accepts the pixel 
information after it has been converted from the video source signal to an acceptable pixel format (e.g. RGB). 
A typical image from the charged-couple device (CCD) television camera is 646 x 486 pixels. That is 313,956 
total pixels to be stored in a file approximately 1.6 Mbytes. Efficient algorithm design is paramount for such 
large unage files. The Personal Ins is a 35 MHz machine that can read an image file and conduct edge 
extraction via the fustredge implementation in approximately 15 seconds (about 3 seconds for an image 


“shrunken” to half of 1ts dimensions). 


B. THE INPUT IMAGE 


Prior to encoding an algonthm to find the straight line edges in an image, a careful analysis of the 
image must be considered. The CCD camera used (JVC model TK870U) can provide a video signal in various 
formats. Some of these video formats include: 


- Betamax Video 
- D1 Video 
- NTSC (National Television Systems Committee format) 
- PAL 
- RGB (red. green, blue attnbutes) 
- Super VHS Video 
There are also various formats for saving and displaying images as files on a computer. Some of these 


formats include: 


- Apple Macintosh MacPaint file 

- Compuserve Graphics Interchange Format file (GIF) 
- Encapsulated Postscript file (EPS) 

- Hierarchical Data file 

- PIXAR picture file 

- Portable Bitmap file 

- Postscript file 

- SiliconGraphics Image RGB file (SGI) 
- Stardent AVS X image file 

- Sun Rasterfile 

- Tagged image file (TIF) 

- Utah Run-Length-Encoded image file 
- Wavefront raster image file 

- XI] bitmap file 


ay 


The RGB format was chosen to be utilized because it is easily translated into NTSC (National 
Television Standard) and exists as a standard format for both the CCD camera output and the SiliconGraphics 


workstation. 


1. The RGB Format 


The RGB format is a 32 bit digital format to describe a single pixel in terms of its basic color 
components (red, green, and blue). In the C language, a 32 bit format is referred to as a “long integer” data 
type. Eight bits are used to represent each of the intensities for the red, green, and blue color components of 
the pixel. This allows for intensity values to range from 0 to 255 decimal (0 to FF hexadecimal) for each 
component. An intensity value of 0 represents zero light intensity for that color component, 255 for full 
illumination. 

The RGB format uses the red, green, and blue colors as the basic components because they are the 
three primary additive colors. An additive color scheme used on television and similar displays combines 
colors from red, green, and blue light sources together to create the rest of the colors in the visual spectrum. 
This is contrasts with the subtractive color scheme implemented with yellow, cyan, and magenta as the three 
primary colors which is used in film photography for filtering a white light source to create the other colors. 

The RGBA format only differs from the RGB format in that it has eight bits reserved for the alpha 
component which is used for transparency on graphics workstations. An alpha component with a value of 
zero is completely transparent, whereas a value of 255 is completely opaque. The RGB and RGBA formats 
are both 32 bits long and are basically identical. The RGB format simply ignores the alpha component and 
operates as if all colors are opaque (alpha value set to FF hexadecimal). Figure 7.1 shows the bit placement 
for the RGB/RGBA formats. In order to allow easier translation between these two formats, the alpha 
component is set to FF hexadecimal. To illustrate this format with some examples, the color black is 
represented as FFOOO000 hexadecimal (alpha = FF; blue, green, red = 00), white is FFFFFFFF, and bright 
kelly green 1s FFOOFFOO. 
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Figure 7.1. The RGB / RGBA formats. RGB format ignores the alpha block. 
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2. Color to Grayscale Conversion 


The findedge implementation of feature extraction requires a grayscale image. However, if a color 
camera is used, it will be necessary to convert the color image to a grayscale image. The RGB format 
describes each pixel in terms of its basic additive primary colors, whereas the overall intensity of the attributes 
would be a grayscale value. To determine the grayscale value, each of the color values must be weighted and 
summed. The weights assigned to each color for representation as a black and white television (grayscale) 
pixel have been standardized by the National Television Systems Committee. The following weights are 


prescribed by the NTSC [HAL89]. 


Red = 0.299 
Green = 0.587 
Blue = 0.114 


Converting a color pixel to a grayscale value is simply assigning the sum of the weighted color 


component values to the grayscale value (equation 7.1). 


R 
RGB, ,ayscale = {0.299 0.587 0.114] |G]. (Eq 7.1) 
B 


4. The NPSIMAGE Data Structure 


The RGB format is a well known and widely used format for defining a single color pixel. 
However, formats for describing entire images via RGB pixels can still vary. One such format, a simplified 
model of a standard video image, developed by Mike Zyda at the Naval Postgraduate School, is the 
NPSIMAGE data structure outlined in the :mage_types.h file in Appendix A. NPSIMAGE is a C language 
data structure (a record type) to store an RGB, RGBA, or color-mapped image. 

For all video images, the pixels are represented as a single, large, one-dimensional array of values 
starting with the first pixel in one corner of the two-dimensional image and ending at the opposite corner 
(Figure 7.2). All pixels in this array are arranged via a single, linear scan of the video image (..e. left to nght, 
bottom to top). In order to properly display the image from a saved format, it 1s crucial to have the dimensions 
of the original image saved. The dimensions of the input image are expressed in number of pixels in the 
horizontal and vertical directions. NPSIMAGE has two integer value slots, xsize and ysize, to store these 


dimensions. 


at 


The image format must subsequently provide an efficient way to access the one-dimensional array 
of pixel values. This is generally done in the C language via a pointer initialized to the first pixel of the one- 
dimensional array. Moving the pointer by altering its offset allows easy access to all pixel values. When using 
the NPSIMAGE data structure, such a pointer will be initialized to the imgdata.bitsptr long integer 


array (Figure 7.3). 


2-D Video Image 


—_______» Fi 








last pixel 


$$» (direction of image scan) 


Figure 7.2. Construction of one-dimensional array of pixel values from two-dimensional video image. 
Order of pixel scan is left to nght, bottom to top, as used in the NPSIMAGE data structure. 


NPSIMAGE *img; /* a pointer to the image data structure */ 


long “ptr; /* a pointer to pixel values 
(represented as long integers) */ 


ptr = img->imgdata.bitsptr; 


Figure 7.3. Declaration and initialization of a C language pointer to the RGB values in an NPSIMAGE. 
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VII. EDGE EXTRACTION RESULTS 


A. FINDEDGE IMPLEMENTATION 

The implementauon of the findedge algorithm described in Chapter V1 and detailed in Appendix B 
provides extracted line-segment features such as in Figure 8.1. The line segments found are from the input 
image of the chair (Figure 3.3). The five parameters for gradient thresholds and line segment properues (C, 
through C,) were determined by trial and error. The effects of modifying the parameters 1s scen in later in this 


Chapter. 





Figure 8.1. Lines found from image of chair via findedge implementation. 
00,000. 0nGa= 20.0 dearces, C, = 0.1, C, = 20 pixels Cs — 20.0. 


La 
Lod 


B. STATIC METHOD OF GRADIENT ANGLE TESTING 

Results from the static method for determination of “closeness” of pixel gradient angle onentation are 
displayed in Figures 8.2 through 8.4. The input image 1s that of the chair (Figure 3.3) and the resultant line 
segments can be compared with Figure 8.1. All five parameters are the same as those used in Figure 8.1. 
Figure 8.2 shows the lines found using the first group of static angles divided by 20 degrees (see Figure 6.2) 
and Figure 8.3 shows the lines from the second group of static angles. Figure 8.4 displays all lines found from 
both Figures 8.2 and 8.3. Comparing Figures 8.1 and 8.4 demonstrates the shortcomings of the static method. 
The line segments are shorter, broken, and subsequently more numerous, all of which are traits not desired 


for the purposes of pattern-matching. 





Figure 8.2. Lines found from image of chair via staric gradient angles method (group | of Figure 
6.2). Cy = 5,000.000.0: C, = 20.0 degrees; C3 = 0.1: Cg = 20 pixels: Cs = 20.0. 
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Figure 8.3. Lines found from image of chair via static gradient angles method (group2 of Figure 
6.2). C1 = 5,000,000.0; C2 = 20.0 degrees; C3 = 0.1; C4 = 20 pixels; C5 = 20.0. 


Figure 8.4. All lines found from image of chair via stanc gradient angles method. 
C, = 5.000.000.0; C, = 20.0 degrees: C, = 0.1: Cy = 20 pixels: C, = 20.0. 


C. FASTEDGE IMPLEMENTATION 

The last algorithm described in Chapter VI, the fastedge method produced lines very similar to those 
found by the fndedge method. Figure 8.5 shows the resultant line segments via the fastedge implementation. 
The major difference between this and findedge was determination of the gradient magnitude threshold value 


C,. Findedge used color to grayscale conversion for an absolute intensity value whereas fastedge only used 


the green color component of the RGB image. 


Peas 
aes 





Figure 8.5. Lines found from image of chair via fastedge implementation. 
C, = 30,000.0; C> = 20.0 degrees; C3 = 0.1; C4 = 20 pixels; C; = 20.0. 
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D. EFFECTS OF EDGE EXTRACTION AFTER “SHRINKING” AN IMAGE 


Condensing an input image to half of its dimensions prior to performing the straight edge feature 
extracion routines produces some noteworthy results. Aside from requiring only a quarter of the 
computational time for edge extraction, the resultant line segments are generally fewer, longer, and seem to 
model the image in simpler terms. To illustrate such differences, Figure 8.6 is resultant line segments (via 
fastedge) of the chair image condensed to half of its length and width (quarter of the original area). Its results 
can be compared with the lines extracted in Figure 8.5. Similarly, Figure 8.7 is a view of a hallway interior 
with its extracted line segments (Figures 8.8 and 8.9) to be compared with the edges extracted from the same 


image after shrinking (Figure 8.10) and edge extraction (Figures 8.11 and 8.12). 





Figure 8.6. Lines found from image of chair after it has been shrunken to half of its 
dimensions and using the fastedge program. C, = 30,000.0; C, = 20.0 degrees; C3 = 0.1; 
Cy = )(} pixels; Cs = 20: 
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interior. 


Figure 8.7. Image of hallway 
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Figure 8.8. Extracted edges of hallway 
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Figure 8.9. Extracted edge 





Figure 8.10. Shrunken image of hallway interior. 





Figure 8.11. Extracted edges of shrunken hallway interior. Edge extraction by fastedge method. 
C, = 30,000.0; C, = 20.0 degrees; C3 = 0.1; C4 = 20 pixels; Cs = 20.0. 
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Figure 8.12. Extracted edges supenmposed over image of hallway interior. 
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E. EFFECTS OF VARYING FEATURE EXTRACTION PARAMETERS 


1. Gradient Magnitude Threshold C, 


The threshold value C, is used to determine if the gradient magnitude of a pixel is sufficiently 


large enough to describe an edge. If the value for pixel gradient magnitude, determined by equation 4.1, is 


larger than C,, then the pixel in the gradient image is set black, otherwise it is set white. In the findedge 
program, the value for C, was determined to be 5,000,000.0 by testing. Figures 8.9 through 8.11 show the 


associated gradient images for values of C, equal to 5M, 8M, and 2M, respectively. 





Figure 8.9. Gradient image of a chair with magnitude threshold C, = 5,000,000.0. 


4) 


eee 
er as 
eee. 


tat pena er ee 4 


. 
. . 
. . 5 
aS, ° 
Pat) 
x Fi 
. 
. 
. 
tee 
o of, 
. 
. 
. 





Figure 8.11. Gradient image of a chair with magnitude threshold C, = 2,000,000.0. 
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2. Gradient Angle “‘Closeness” Angle C, 


The threshold value C,, the maximum difference between two pixel gradient orientations was 
tested by trial and error to produce more extracted line segments that best represented the straight edges from 
the input image. For the umage of the chair (Figure 3.3), C, = 17.2 degrees performed best. Larger angles 
resulted in combining multple straight edges from the image into one extracted line segment, whereas smaller 
angles were too restrictive to produce long line segments. The best C, value for one image is not necessarily 
the best value for all images. Pictures of the hallway interior, used for pattern-matching and pose- 
determination, utilize C, = 28.65 degrees (0.5 radians). The images in Figures 8.12 through 8.14 illustrate the 


effects of modifying C, of the chair image via the fastedge implementation. 





Figure 8.12. Lines extracted from image of chair for C) = 17.2 degrees (0.3 radians). 
C; = 30,000; C3 =i): I; C4 at) pixels; Cs = 20.0. 
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Figure 8.13. Lines extracted from image of chair for C, = 30 degrees. 
C, = 30,000: C, = 0.3: C4 = 20 pixels: Cs = 20.0. 


Figure 8.14. Lines extracted from image of chair for C, = 12 degrees. 
C, = 30.000: C3 =o Ci = 2{) pixels: Cs = 200: 


3. Line Test Parameters C3, Cj, and Cs 
The three line test parameters identified in Chapter V are utilized for limiting the number of 
extracted line segments by placing restrictions on the ratio of axes lengths (Pp = dninor/Amajor)» the number of 
pixels (77g), and the major axis length (d,,,;0,), respectively. The lines extracted from the chair image by 
modifying C3, C4, and Cs are shown in Figures 8.15 through 8.17. Each Figure can be compared with the 


baseline extracted features in Figure 8.12. 





Figure 8.15. Lines extracted from image of chair for maximum thinness ratio C3 = 0.05. 
C, = 30,000; C2 = 17.2 degrees; C4 = 20 pixels; C5 = 20.0. 
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Figure 8.16. Lines extracted from image of chair for minimum number of pixels C, = 40 pixels. 
C, = 30,000; C, = 17.2 degrees; C3 = 0.1; Cs = 20.0. 





Figure 8.17. Lines extracted from image of chair for minimum axis length Cs = 50.0. 
C, = 30,000; C> = 17.2 degrees; C; = 0.1; Cz = 20 pixels. 
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IX. MATCHING LINE SEGMENTS 


The focus of this Chapter is the development of a matching algorithm that compares the straight edges 
extracted from the input image with the linear features of the environment model. The pattern matching of 
two-dimensional line segments can be divided into two different problems: 


1. matching with the model based upon a known location and orientation of the robot (the pose 
or configuration of the robot), 
2. and matching with the model without knowledge of the robot’s pose. 


The first problem is applied to position correction and vehicular navigation, whereas the second 
problem is that of object identification and viewing aspect determination. The first problem is much more 
constrained than the second and is pursued herein towards the goal of pose determination and vehicle 
navigation. Expecting small dead-reckoning errors to anse in the vehicle’s pose, the matching algorithm must 
exhibit some degree of robustness if it is to make valid corrections. Therefore, the goal of the matching 


process 1s to find the best match of image lines to model lines. 


A. THE ENVIRONMENT MODEL 


The testing and operating environment for Yamabico- 11 is the fifth floor of Spanagel Hall at the Naval 
Postgraduate School. Modeling this environment in a graphical database [STE92] was accomplished in 
conjunction with the above edge extraction methods for the pattern matching task described in Chapter IX. 
The model ts a “2-D +’’ model similar to that described by David Marr [MAR79]. The environment model is 
designed to support requirements for visual navigation, sonar navigation, shortest distance path 
determination, and safe path planning. The floorplan features are expressed in the xy cartesian plane and 
associated heights of hallway features are modelled in the z axis. The notion of free space is aided by defining 
polygons in the xy plane as either floor or ceiling polygons. All database measurements are in inches. Ail 


support routines for implementation of the model are coded in ANSI C and described fully in (STE92]. 


1. Interfacing the Model Database 
Since the application features of the environment were encoded in the same language as the image 
processing and pattern matching methods, accessing the model is quite simple. The pattern matching routine 
will invoke calls to both the image processing and environment model functions to access the line segment 
primitives utilized in the matching process. The primary call to the model (get_view()) requires providing an 


estimated pose (position and orientation information) of the robotic vehicle and the focal length of the camera 
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lens. The function returns a data structure that includes pointers to linked lists of the line segments of the 
model database mapped to a two-dimensional viewing plane that the robot should expect to see given the 
input pose and visibility constraints of occluding edges. The line segments shown in the model’s two- 
dimensional view represent the orthogonal features of the operating environment, primarily the junctures of 


the walls, ceiling, floor, doors, and overhead lights. 


B. BASIC COMPARISON OF LINE SEGMENTS 


Since the environment model provides a two-dimensional view of the known linear features for a given 
pose, the matching algorithm needs only to determine linear matches in two dimensions. From the results seen 
in Chapter VII, the extracted edges from the input image do not cross, overlap, or touch. The extracted linear 
edges are, at best, incomplete segments of the modelled linear features. The matching algorithm must be 
based upon individual matches between the image's extracted edges and the model's linear features. When 
comparing two-dimensional line segments, three aspects must be considered: translation, rotation, and scaling 


differences (Figures 9.1 through 9.3). 


Image line 
d2 







model line 


translation distance, d 


Figure 9.1. The translation distance, d, between two lines. 
Can be computed as the median between distances d/ and 
d2, perpendicular to the model line. 


Image line rotational difference, o 







model line 





Figure 9.2. The rotational difference, a, between two lines. 
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image line model line 










length of image line mapped onto model line 


scaling factor, $ = 
length of model line 


Figure 9.3. The scaling factor, s, between two lines. S is the ratio of the image line 
mapped onto the model line with the model line itself. 


An additional requirement based upon the properties of the extracted edges using least squares fit can 
also be stipulated. That is, the endpoints of the image edges must lie within the endpoints of the modelled line 
segment. This check for endpoint inclusion may be relaxed to allow for vehicle translation errors. This is done 
by keeping the difference that the image edge endpoints lie outside of the endpoints of the modelled line 
segment, divided by the length of the image edge (d,,4jo,), less than or equal to a specified amount (k in Figure 


9.4). 


Image line 1 model line 





image tine 2 
(length = 7) 


Figure 9.4. The inclusion of image endpoints within the model line’s 
endpoints. Image line 2 can meet this requirement if (A/ I< k, (0.0<k <0.1). 


Another consideration for matching image lines to model lines is that since the extracted image lines 
are often incomplete and broken segments of a modelled feature, many image lines may be matched to one 
model line as shown in Figure 9.5. The only stipulation is that the image lines do not overlap when mapped 


onto the same model line. 
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model line 
image line 2 







Image line 1 Image line 3 


Figure 9.5. Many-to-one relationship of image lines to model lines. 
Image lines 1, 2, and 3 may all be matched with the model line. 


These five aspects of two-dimensional line matching could be combined to produce one value to 
describe the confidence for the match of one image line to one model line. Summing individual matches with 
highest confidences provides the most likely matching between image edges and model line segments, yet it 
remains to be determined if it is the correct matching. Beveridge [BEV90] includes an error term for model 
line segments that are omitted from the matching process. Many graph-based matching algorithms outlined 
by Ballard [BAL82] are NP-complete problems. Beveridge’s method defaults to continually permuting 


various matching combinations in order to find the optimum matching. 


C. MATCHING IMPLEMENTATION FOR VERTICAL LINE SEGMENTS 


Assuming that the robot’s camera is in a fixed position and that the robot ts on a stable platform, the 
pose (or configuration) of the robot can be described in three degrees of freedom (x9,¥9,09). With this 
assumption, all vertical lines in the real world will be vertical in any image. Therefore, matching only vertical 
lines should provide a simple solution for the pose determination/correction problem. Since all line segments 
will be vertical, rotational differences between the image lines and model lines may be neglected, and the 
translational distance, d, can be simplified to be the horizontal angle from the midpoint of the image line to 
the model line. 

Considering an image plane M that exists at focal length f from the center of the lens system of the 
robot’s camera, the vertical edges extracted from the image will appear as in Figure 9.6. For a given estimated 
pose (vy.¥p.89). an Overview of the robot and the image plane M (Figure 9.6) will be situated in the operating 
environment model as in Figure 9.7. The environment floor is the vy plane and heights of modelled features 


are expressed in the z-axis. Image plane M is parallel with the z-axis. 
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image plane M 





Figure 9.6. The image plane M. The image is centered on image axes u,v with three 
extracted vertical lines at u;, uz, and u3. 


3 image lines at 
Uy, Us, and U4 
image plane M 







tana=u/f 


camera’s (robot's) pose = (Xp, Yo, 99) 





Xx 


Environment coordinate system 


Figure 9.7. The robot in the environment coordinate system. The robot’s pose is defined by (x,. y9.8p). 
The image plane M exists at the focal length (f) from the robot’s camera. Vertical lines extracted from 
the input image in Figure 9.6 (at image u-coordinate positions u;, u, and u;) can be described by an 
angle a from the center of the image. 
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A two-dimensional view of the environment model is constructed given the three-dimensional model, 
an assumed pose, and the focal length f (Figure 9.8). The assumed pose is provided by the robot’s dead 
reckoning capabilities from wheel motion and is used knowing that it is an approximation. Therefore, the 
matching problem to evaluate how well the image edges from the image plane M fit onto the model lines in 
the two-dimensional plane M’. For this implementation using only vertical lines, quantifying the match 
between an edge and a model line will be based only upon the translational distance d, if the endpoint 


inclusion requirement is met. 






3-D Environment Model 






Image Plane M’ 
2-D View of Environment given Robot’s pose 
Xx 
Robot’s Pose (x, y, 9) 
Figure 9.8. The image plane M’. M’ is the two-dimensional view of the modelled environment 


for the given pose (x9,¥9,99) of the robot (camera). M’ can be thought to exist at a distance equal 
to the focal length, f, of the camera’s lens system. 


The line matching provides connectivity between the extracted image lines and the vertical features of 
the environment model. Thus, the three most significant extracted image lines can be matched with three 
vertical model lines as in Figure 9.9. Vertical image lines u;, u2, and u;, from the image plane M, are best 
matched with vertical model features L1, L2, and L3, respectively. This does not ensure that this is the correct 
matching, but it will provide a means to calculate a possible pose for the robot. This method also assumes that 


the three most significant vertical image lines are produced from modelled linear features. 


a2 


Since backtracking may be necessary for recovering from successive attempts in the pose 
determination algorithm, every image edge in M must be able to support a linked list to all of the possible 
matches to the model line segments in M’. Appendix C contains the code for this implementation, titled 
vertmatch. A data structure named MATCHTYPE in the file match_types.h (Appendix A) is used for the 
association of an extracted edge with a model line segment. Each image edge data structure has a pointer to 
the head of the list of possible MATCHTYPEs and a separate pointer for the best. The different possible 
matches of an image edge to various model line segments are ordered by horizontal angular difference 
between the location of image edge and model line. The smallest difference, that being the best match, is at 
the head of the match list and all subsequent matches follow. The pose determination/correction algorithm in 
Chapter X utilizes only the three most significant extracted image edges (i.e. the three image lines with the 
greatest values for d,,;.,). Thus the vertmatch implementation creates a list of MATHCTYPEs for the three 


most significant edges. 


L3 * modelled vertical feature at (x; 3, ¥;.3) 






Image plane M 
with vertical lines 


wy = atan2(y; 3 - Yo, Xi3 ~ Xo) 


L1 


Camera's pose = (Xo, Yo, Qo) 





Environment coordinate system 


Figure 9.9. Modelled vertical features matched with extracted vertical edges from an image. L1, L2, 
and L3 are the labels assigned to the vertical lines matched to the three most significant vertical 
image lines in found in M. Angle y is the orientation of the line from the camera’s position to the 
location of a vertical line. The modelled vertical line should then appear at an angle (89 - w) from 
the center of plane M’, the 2D view of the model environment (Figure 9.8). 
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X. CORRECTING ROBOT'S POSE 


With the assumption that the robotic vehicle is a stable platform with its camera at a fixed position, 
there are only three degrees of freedom in the model space (x, y, and ® - corresponding to the variables of the 
robot's pose) that need to be determined. If considering only vertical lines for matching, it is possible to 
determine the camera’s pose from only the three most significant vertical lines in an image, provided that they 


are all produced from features represented in the three-dimensional environment model. 


A. POSE DETERMINATION 

The only known, accurately measured quantties that are available for pose determination are the -ry 
coordinate map locations of the three vertical model lines and the horizontal angular differences between the 
extracted vertical edges u;, U2, and u; from Figure 9.6. Let us denote two of these angles, the angle between 
the leftmost (u,) and middle (u») image lines and the angle between the middle (u>) and nghtmost (uz) mage 
lines as o and 8. respectively. If the best matches for u,). U2, and u3 are used (vertical model lines LI, L2, and 
L3), angle o. is the expected viewing angle between LI and L2, and B is the expected viewing angle between 
L2 and L3. There is exactly one location that will have viewing angles a and B between model lines L1, L2, 
and L3. If the distance between two vertical model features is thought of as the side of a tnangle (a) and the 
associated viewing angle (a) as the opposite angle of the triangle, an infinite number of possible locations for 
the viewing angle exist on a circle that circumscnibes all possible triangles (Figure 10.1). For two triangles, 
one with side a and opposite angle a, and the other with side b and angle B, two circles of possible viewing 
locations can be constructed as in Figure 10.2. Sides a and 6 are therefore chords of the two curcles. There are 
two intersections of the circles, provided that the circles are not identical. One intersection is at the location 
of vertical model line L2 and the other is the possible position of the camera (Xposs: )'poss) The camera must 
be at this position for this line matching, given the viewing angles a and B between the three major vertical 


SF 


WS 


Figure 10.1. Triangles with side a and opposite angle a, circumscribed by a circle. 





(Xposs-Y poss) 
possible correct robot position 


Figure 10.2. Geometry for ie determination. Given measured locations of vertical features 
in the environment model (L1, L2, and L3), circle chords a and b can be calculated. Angles 

a and f are measured from the image plane and are defined by the angular difference between 
the vertical extracted edge features from the image. With chords a and b and angles « and B, two 
circles can be constructed such that their intersections will exist at the position of model line L2 


(¥,.2.L2) and a possible robot position (%,,555:)’poss): 


Once a position has been determined, calculation of the camera’s orientation in the environment is 
simple. Since the locations of the vertical modelled features are known in the environment map (1, ;, yj), the 
angular onentation, y, from the possible position of the camera (xpos) Yposs) iS ataN2(x1j - Xposs> YLi - Yposs)- 
The angular location of these vertical model lines from the vertical centerline axis in the two-dimensional 
plane M’ is 0, — y. Averaging the differences between angles of the model lines from the center of the plane 
M’ (8. — wy) and the angles of the image lines from the center of the image plane M provides a rotational 
correction. The correction 1s added to the orientation of the input (dead-reckoning) pose, 9p, to get the 
possible pose orientation, 0,,,,. The pose for the camera based upon the given matching is therefore 


determined (Xposs> ¥poss: Sposs)- 
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B. POSE VERIFICATION 


The next best match combination will be performed by selecting the next match with the least 
horizontal angle difference between the vertical image edge and model line segment out of all three image 
line match lists. For this new combination of three matches, another possible pose is calculated by the above 
method. This is done for the 7 best match combinations. An exception to this rule is to skip matches where 
two or more image lines are matched to the model line segment. This method will provide, at most, n possible 
poses from the 7 match combinations. 

Since a unique pose can be determined from any given matching combination for the three most 
significant image edges, the determination of the correct pose of the robot must rely upon a means to verify 
the pose and the matching combination. This topic is descnbed by Heller and Stenstrom [HEL89]. For a given 


possible pose (x ), locations for all the model line segments can be calculated if they were to 


poss: poss Oposs 
be viewed in the image plane M. Comparing all of the extracted image lines to see how many fit the 
repositioned model lines provides a means to evaluate the possible pose. However, since many image lines 
are not produced from modelled features using the number of good image to model line fittings alone can not 
guarantee the correct pose. 

If some credibility is expected of the estimated pose for which the two-dimensional view of the model 
(M")was constructed from, then calculated possible poses should be close to the estimated pose. Thus, an 


evaluation (eval...) combining the number of image edge to model line segment fittings ( ) with the 


poss Mposs 


distance between the possible pose and the estimated pose (d,,,,) will be of the form 


OSS 


m 
eval joss = aia (Eq 10.1) 


poss 





The possible pose with the greatest value for (eval...) will then be considered the corrected pose. 


lposs 


Results of this method are shown in Chapter XI. 
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XI. PATTERN MATCHING AND POSE DETERMINATION RESULTS 


The results from three tials are shown in this chapter. The actual camera pose is measured from a 
reference point in the operating environment The estimated pose for developing the two-dimensional view 
of the model environment was encoded in the program vertmatch to simulate dead-reckoning errors. Each 
trial has a series of four figures. The first figure is the input image with the extracted vertical edges. The 
second figure is a wire frame two-dimensional view of the environment model based upon the estimated pose. 
The third figure is the wire-frame model superimposed over the input image. The fourth figure is a wire-frame 
model based upon the corrected pose superimposed over the image. The inputs to the vertmatch routine are 
the image, the environment model, and the estimated pose. Output from the vertmatch routine lisung the 


matches and possible poses is shown between the third and fourth figures of each mal. 


A. TRIAL 1 


e722 
¢ 
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Figure 11.1. Trial 1, input image with extracted edges. 
Actual camera pose: Xp = 60.0 inches, yo = 366.0 inches, 89 = 253.0°. 
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Figure 11.2. Trial 1, 2D wire-frame view of model based upon estimated pose. 
Estimated pose: Xo = 60.0 inches, yo = 366.0 inches, 09 = 250.0°. 
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Figure 11.3. Trial 1,2D wire-frame view superimposed over input image. 
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Scaipe stabted on Thu’ Mar 19 12:12:18 1992 
tUrminGgulreventmaten 060366253.pic 
vertmatch> 060366253.pic xsize= 646 ysize= 486 pixels= 313956 
lines found in image written to: ‘lines.text’ 
Number lines found in 060366253.pic = 17 


DEO Inpue) pose; x — 60.00, y = 366.00, theta = 250.00(-1.92 rads) 
Nr Vertical Model Lines = 20 


Determine 3 major image lines for matching: ----- 


1) line 13, angle from center = -0.1819 
2) line 10, angle from center = 0.0980 
3) line 8, angle from center = -0.0814 


Image line matchlists to Model line NAME(angle difference): ----- 

left (edge 10) > 

N(0.0526) M(-0.0634) L(-0.0675) I(-0.0904) J(-0.0904) H(-0.1279) 

Oren (Oey 3/) E(=022176) D(-0.2224) C(-0.2288) A(-0.2300) 


middle (edge 8) > 
Peo OCS En 0.20382) )(-050430), G{0.0473) C(-0.0494) A(-0.0506) 
ego Peso 0890) 1 (0508690) (0.1119) M(O.1160) N(0.2320) 


right (edge 13) > 
mona 98)  CrOe Oot) D(0.0575) E(0.0623) F(0.1062) G(0.1478) 
Heomle 20) deb 95) N(0..3325) 


Pose determination: ----- 


[ N, F, A] x= 55.42 y= 513.24 T= 147.3124 R= -0.3730 nr verify matches= 


Pao nol seOok MODIFIED *** 


[ N, E, A] x= 276.04 y= 256.66 T= 242.1306 R= -0.6891 nr verify matches= 


ion Dee eNeepOsituon: chord length < MIN CHORD LENGTH 


[ N, G, A] x= 51.84 y= 381.92 T= 17.8955 R= 0.0048 nr verify matches= 


Bases l POSE SMODIFIED *** 

C, A) NO position: chord length < MIN CHORD LENGTH 

A, A] NO position: 2 IMG LINEs matched to same model LINE 
eine posteden: cChond lengeh < MIN CHORD LENGTH 


™~ 
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*QZ2az 


td 
Bool POSE MODIFIED ==* 


a a) ee reees, ee —- Sor 
pa 
Ss 2S 
io i 7 
~ ~ 
(Sa 
~ 


El) NO position: chord length < MIN CHORD LENGTH 
PeeieNOeposit lon. ichord length < MIN CHORD LENGTH 
Pee Oepeaiticon. 2 0lMG LINES =matened to same mode] LINE 
PeLleNe position. chord length < MIN CHORD LENGTH 
F] NO position: chord length < MIN CHORD LENGTH 
mee) NO@ position: chord length < MIN CHORD LENGTH 
mee NO position; chord lengehn < MIN CHORD LENGTH 


~ ~ 


™ 


~ 


BPaauauagayHete 
ZO WWH a 


~ 


V— 


[ G, M, F) x= 270.77 y= 246.45 T= 242.3155 R= 0.8120 nr verify matches= 
WcepeGieNG position: 2 IMC LINEs matched to same model LINE 

[ G, M, H] x= 307.93 y= 268.25 T= 266.5063 R= -1.4201 nr verify matches= 
[ F, M, H] x= 340.40 y= 274.86 T= 294.8414 R= -1.5850 nr verify matches= 
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H, C) x= 58.49 y= 368.04 T= 2.5417 R= 0.0464 nr verify matches= 


H, D]) x= 58.48 y= 379.09 T= 13.1814 R= 0.0087 nr verify matches= 
H, E) x= 86.10 y= 459.31 T= 96.8872 R= -0.2843 nr verify matches= 
H, E]) x= 307.54 y= 268.49 T= 266.0523 R= -1.1971 nr verify matches= 
H, E}) x= 307.41 y= 268.42 T= 265.9568 R= -1.1694 nr verify matches= 


M, F) x= 276.86 y= 247.81 T= 246.9754 R= 0.8255 nr verify matches= 


oOo nN ~ 


N NO 


oO © 


[ F, M, J] NO position: chord Tengthe< MINNCHCRD LENGTH 

[ E, M, J] N@s position: chord length] < (MINse@RORDeeEn ora 

[ D, M, J] NO position: chord length < MINGCHORD LENGTH 

[ C, M, J] NO position: chord length < MINV@HORDMEENGTaA 

[ A, M, J} NO position: chord length < MIN CHORD LENGTH 

[ A, N, JJ] x= 337.19 y= 307.83 T= 283.2245 R= -0.5412 ne veri & matches— 10 
[ A, N, N] NO position: 2 IMG LINEs matched to same model UINE 


Corrected pose: x= 58.49, y= 368.04, theta= 252.66(-1.8735 rads) 
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Figure 11.4. Trial 1, corrected 2D wire-frame view superimposed over input image. 
Corrected pose: x9 = 58.49 inches, yo = 368.04 inches, 69 = 252.66°. 


Translational error from actual pose = 2.54 inches, rotational error = 0.34°. 
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Figure 


Bemeee Stdtteo Of wal Mar 19 12:19:42 1992 
Bering 1s vercraccr 059366249 .pic 
Ne Geiger e249 ple xsize= 646 ysize= 486 pixels=. 313956 


Pie Oo NOu ee mace written to: ‘lines. text’ 
NMC eee eerne 1m 059366249.pic = 34 


Poe.) cese. x =) 60,00,-y =) 366.00, theta = 250.00(=1.92 rads) 


Ne wverecca! “Mocel Lines .=*20 


Dete wee Geec” (mage lines for matching: ----- 
[Meese 2 , eee from center = -0.1113 
Zire 6 26, .arc-e Zrom center = 0.1663 
See GC eo aoe from center = 90.0509 


iii=Ges ese a 7e-2is~s to Mccel line NAME (angie cifference): ----- 
Ve Se es Zein 


Pesaweeeon er to BS) (0. 1358) 11-6 .258 7) ie hoon 0. 1967 ) 
ei rs oe ee Oe Oe (moo OC (— 6 7 O71) eA -0.2983) 
iii 1S lan (ease ) > 
PiG= oko 6.4) oe ee ee OOo (—-C 0 4iss) a i-7 -0S0e) Gi(-0.0850) 
So Soe) (- Pees oO 200) oC ides) 2 f—-0- 1817) Al(-0.1829) 
weg ee (SCCeo ks, . | 
EV. 22) fem) 0 9a ee 707) eC ..0357) Gi0.0773) 
Ge ee ieee ae Pee OO) oe eee 2) VCO. le 60) N(O.2619) 
Io seece ler el an! aS 
ee ee Se Se oe 0544 nr verify matches= 10 
—— sss) eee ae Fe 
ee ee ee 0.6 f= 4 Ses 2-5 US. nr verify matches= 1? 
Sa ee: Oe ee OOo ee 7 NY Verifyematches= 16 
ome aS ie ee, 
Pee eX ey 9 ae ee 496 NY verify matches= 15 
fone oe ey ee ee 4 SCOOT verify matches= 15 
7 . Oe ee ee - - eee NY verify matches= 
i ae eee Y= 6 56 et l= 2 oGe 6. x= —5. 75.4 nr verify matches= 9 
Nee ree by 48 er l= 497606) x -0.7514 nr verify matches= 9 
eee eso) «CULV = 846.22 T= 206.2917 =p. 8209 nr verify matches= 8 
eee Ss 6 emer Ons ChCre .enrgcn < MON <HORD LENGTH 
Pees ee Seen, 2 IMG LINES macenec to same model LINE 
\, ae Be S220ons Charo lenci 4 enon ENGTH 
De ee Ce 2 EMC LINES el lenee =o Same model LINE 
ae Bee | eee ee, YG Noe. cee eco Same model LINE 
ee eee ee oe On: 2 MG INES macenea to same model LINE 
eee es 2emee. Y= 215.99 T= 25276925 -8= 1.9803 nr verify matches= 7 
Doe ee Iax= $27,9. y= 259.29 T= 274.5549 x= -0.3348 nr verify matches= 0 
a. x eee = 2 oe = 29 ee ee ee -0.5230 nr verifyematches= 0 
ee 5 1On: 2 MG Lines maccrnec ©£cC same: model LINE 
Ve ween 52. 50 Y= 209.25 T= 2)/2,8296 R= .2428 nr verify matches= 0 
Pee (eX Sook Y> 257.00 T= 27023404 R= -¢.3378 nr verify matches= 0 
Bee SS YW 297, 00 T= 27075404 R= -0.3378 nr verify matches= 0 
eee ee ee = 257, Oo = 249.2 72 R= -).4951 nr verify matches= 0 
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Figure 11.8. Trial 2, corrected 2D wire-frame view superimposed over input image. 
Corrected pose: \9 = 58.73 inches, yg = 366.45 inches, 89 = 249.02°. 


Translational error from actual pose = 1.35 inches, rotational error = 0.02°. 
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Figure 11.9. Trial 3, input image with extracted edges. 
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Oe a Oa 30.0605) A0(-0 05616) AN(-0.0625) AM(-0.0647) AL(-0.0654) 
PEO. Joos (-0. 2422) M(-0. 1442) L(-0.1490) K(-0.1506) J(-0.1614) 
ee ye tooo) GC (=-072049) F(-0.2052) D({-0.2743) C(-0.2863) 
B(-0.2981) 

Pose determination: ----- 

[BC,5C,BA] NO position: 2 IMG _LINEs matched to same model LINE 
[BC,BC,BB] NO position: 2 IMG _LINEs matched to same model LINE 

pene eee eNO Positacen: 2 IMG LINES matehed to same mode] LINE 
eG oe 2 el emo st (1007 IMG LINES matched to same model LINE 

PBC, BC, AY] "N©=oos-tion: 2 IMG LINES matched to same model LINE 

[BC,BC, AX; NO position: 2 IMG LINEs matched to same model LINE 
eee ee 2 ey 7377237 T= 462.9611 R= 0.9733 nr verify matches= 0 
Poe est POSE MODIFIED *** 

[BC,38,AW!] x= -2.16 y= 734.34 T= 460.0836 R= 0.9748 nr verify matches= 0 
Oe e7sn, 6 xX 8763 y— 619.54 T= 344.7975 R= -0.0743 nr verify matches= 12 
Peeioe ol Ee CSr MODIFIED *** 

{BC, 3A, AV] x= 7.98 y= 623.75 T= 349.0543 R= -0.0781 nr verify matches= 12 
ee Xe Y— 024.25 T= 349.5528 R= -0.0788 nr verify matches= 12 
ee xe eG Y= 623,89 T= 349.1925 R= -0.0795 nr verify matches= 11 
BC,5A,AS! x= §.04 y= 623.33 T= 348.6295 R= -0.0795 nr verify matches= 11 
(BC,AZ,AS, x= 69.07 y= 298.72 T= 21.7457 R= -0.C466 nr verify matches= 13 
Meese s le Ose VOOLF IED *** 

eee |X SG S2 y= 354.76 T= 78.4074 R= -0.0593 nr verify matches= 15 
(BB,AZ,AaR. x- 3€.98 y= 362.97 T= 86.6773 R= -0.0600 nr verify matches= 16 
Reo ee tk 26. 20 Y= 372-6467 T= 96.3157 x= -0.0632 nr verify matches= 16 
'BB,AY,SO_ x= 67.12 y= 172.40 T= 106.3377 R= -0.0423 nr verify matches= 15 
[BB,AY,A42. X= 50.78 y= 240.96 T= 36.1426 R= -0.0580 nr verify matches= 14 
(BB,AY,AO x= 4€.30 y= 252.24 T= 24.7655 R= -0.0606 nr verify matches= 14 
ee ex eee ee Y= 260.59 T= 16.6734 x= —C.C625 nr verify matches= 15 
[eee 5. SCS wow lob *** 
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BA,AY, 1 x= 2;,.20 y= 452.9€ T= 177.1836 R= -0.0741 nr verify matches= 14 
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XH. CONCLUSIONS 


A. FEATURE EXTRACTION 


The edge extraction implementation fastedge, provides linear features that, unlike the Hough 
transform, have good endpoint information. The implementation requires approximately 15 seconds to 
extract the linear edges from a 486 x 646 pixel image (313,956 pixels total) on the Personal Iris (35 MHz 
clock cycle). For a picture that has been “shrunken” to half of its width and height, the process requires only 
3 seconds. 

Selecting the appropriate constants C, through C; is important. The effects of modifying these 
parameters, outlined in Chapter VII, provide optimum values for use in a specific environment. The task of 
determining optimum values for a dynamic environment could be automated; however, it would require an 
extra scan through the image or else added hardware (e.g. a lightmeter). 

The version implementing two static sets of angles used for the description of pixel gradient 
orientations could provide a very fast hardware implementation. Although it did not perform as well as 
averaging gradient orientations, the speedup advantages gained from implementing this method warrant more 


consideration and experimentation. 


B. PATTERN MATCHING 


Pattern matching is the “weak link” in the vertmatch implementation. Although the method is quick, 
it is crude. It was chosen as a means to provide a simple basis for the pose determination problem, therefore 
opting for the easier problem of matching only two-dimensional vertical line segments with the assumption 
that the three most significant edges extracted from the image are products of vertical modelled features. 
Many line matching implementations are graph based and therefore rely upon links between the extracted line 
segments that can only be inferred, but not accurately determined. Graph based matching algorithms are often 
NP complete thus leading to alternative methods like Bevendge’s randomly permuting matches to find the 


optimum match. 


C. POSE DETERMINATION 


The pose determination algorithm was based upon simple vertical line matching in a known orthogonal 
environment and the geometry for its foundation allows a unique pose to be calculated simply and quickly. It 
was found to provide good estimates from three visual lines of bearings. Small errors between the actual and 


corrected poses in the three trials in Chapter XJ could be from the algorithm, or from hardware floating-point 


69 


constraints involved with the trigonometric functions, or (most likely) from data measurement error from the 
tape measure. 

The pose determination test trials were conducted at two different primary locations, viewing the 
elevator alcove and viewing the long axis of the hallway. Trials 1 and 2 in Chapter XI were conducted with 
the environment features of the elevator alcove approximately 20 feet from the camera which provided only 
20 vertical model line segments. With relatively few model line segments for the three major image edges to 
be compared with, the matching was easy to check. Trial 3 involved much greater distances and over 100 
vertical model line segments. The corrected pose could be more accurate if more iterations for possible 
matches and poses are conducted. However, the actual pose can be determined if all match combinations are 
tested exhaustively. Implementation of backtracking with some good heuristics about the matching process 
would be beneficial here. The long aspect of the hallway for pose determination is more difficult in terms of 
distances and possible matching combinations; however, other features (edges from baseboards and overhead 
lights) may be utilized to enhance the performance. Thus, a more general and more robust line matching 


implementation is required. 


D. FOLLOW-ON WORK FOR YAMABICO-I1 


The methods developed in this thesis only have been directed towards the first objective of visual 
navigation in a known environment and has not dealt with the greater complexities of object recognition in 
an unknown environment. There remains much work to be done for continuing the implementation of visual 
navigation capabilities on Yamabico-11. Some of these areas are: 


Coding the methods in this thesis in 68020 microprocessor assembly language. 

Developing a vision based support programming language similar to the Mobile Motion 
Language. 

Implementing the environment model support routines in hardware. 

Developing a multiprocessor architecture to handle image edge extraction, two-dimensional 
environment view construction, line matching, and pose determination in parallel. 

Incorporating sensor fusion of sonar data with the vision navigation routines to check accuracy. 

Develop and implement a fast, robust line matching method for all lines. 

Develop a method to positively determine if an edge is from a modelled linear feature. 

Develop object-based recognition methods to handle known and unknown objects. 

Develop an vision based algorithm that supports a reflexive behavior for obstacle avoidance. 

Develop methods to track moving obstacles. 
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APPENDIX A - DATA TYPES 


FUNCTION: ATAN2 


Oe aS ce ge a ER Ra I OR AR I HK eH KK Re OK KH 
/* file: atan2.c 

i 

/* Provide a definition of the atan2() function. 

* 


ee ee ee OES FERS EEE EEE EE ARERR AE ER EAE TE A EEE | 


#define PI 3.141592653589793 


double atan2(y,x) 
| double x, y; 
If (x > 0.0) return (arctan(y/x)); 
else if ((x < 0.0) && (y > 0.0)) return (arctan(y/x) + PI); 
else if ((x < 0.0) && (y < 0.0)) return (arctan(y/x) - PD); 
elsetincx — 010) & & (y == 0.0)) retum (PI); 
else if ((x == 0.0) && (y > 0.0)) return (PI/2.0); 
else if (x == 0.0) & & (y < 0.0)) retum (-PI/2.0); 
else return (0.0); 
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B. 


DATA TYPE HEADER FILE: IMAGE TYPES.H 


[PRR EER ARE EE EB EE IR IIE EE 
/* file: image_types.h 

* 

/* This file holds the structure definitions for the different 

/* types of in-memory images. 


/* Types defined: CMAPIMAGE 
ie NPSIMAGE 


ee 


#define RGBA 1 /* RGBA 24 bit images (alpha 1s Oxff filled) */ 
#define CMAPPED 2 /* color mapped images */ 
#define RGBAWITHALPHA 3 /* RGBA 32 bit images where alpha is read/saved 
in the image files. */ 
/* define a structure type for color mapped images */ 
struct cmapimage 
short *bitspu; /* the bits for the short images */ 
long nentries; /* the total number of entries in the color map */ 
short *rcds; /* ptr to the red entries of the color map */ 
short *greens; /* ptr to the green entries in the color map */ 
short *blues; /* pur to the blue entries in the color map */ 
long cmapofttsct; /* color map offset, 1.c. the first color we will us in the color map */ 
wn struct cmapimage CMAPIMAGE: /* define a CMAPIMAGE type */ 
/* define a union so that the top level image structure’s last pointer can point to several different 
kinds of images.” / 
union imagedpu 
long *bitspur; /* long images need no more data than a ptr to the bits. */ 


CMAPIMAGE *cmapptr; /* a color mapped image must have the bits and a color map so 
we need a complete structure. */ 


/* define the top level structure for the image */ 
struct image 
| long type: /* image ly pe */ 

long XSIze57" ASiZC Ol In@ mace +) 

long ysize; /” ysize of the image */ 

char *namce; /* ptr to string naming the image */ 


union tmagedptr tmgdata; /* ptrs to data for this type of image */ 


js 
typedef Struct image NPSIMAGE: /* define an NPSIMIAGE type | 


(ke 


DATA TYPE HEADER FILE: MATCH_TYPES.H 


et Re TEASE EASELS TEETH TESLA ESLER EY AEE EE EES SLES ES HS HS Ht HO 


/* file: match_types.h 
* 


/* This file holds the structure definitions for image edges and 
/* match types to model lines. 
* 


/ 
/* Types defined: EDGE 
jf 


POINT 
a POSE 
ih MATCHTYPE 
: IMG_LINE 


i ee PTET EEE TEESE SEES EET ESE A EERE EES ES | 


typedef struct edge_region_type 


int active; /* boolean if past region is appended to a present region */ 

long first_pixel; /* first and last pixels added to region */ 

long last_pixel; 

long xmin; /* min & max pixels for previous row */ 

long xmax; 

double avg_phi; /* for use with dynamic averaging of gradient orientation, phi */ 
double sum_phi; 


/* Least Squares Fit momments: */ 
long mO00; /* Number of pixels */ 


double m10; /* Sum x */ 

double m0}; /* Sum y */ 

double m11; /* Sum x*y */ 

double m20; /* Sum x*x */ 

double m02; /* Sum y*y */ 

struct edge_region_type *next; /* ptr to the next EDGE */ 
HEDGE: 


struct point_type 
{ 
double x,y; /* x,y coordinates of the pixel endpoints */ 


}; 
typedef struct point_type POINT; 


ye 


struct pose_type 
{ 


float x, y, theta; 


i. 
typedef struct pose_type POSE; 


typedef struct match_type 
{ 


LINE *line; /* pur to LINE type (from Jim Stein’s “graphics.c’) */ 

double angle_view_diff; /* angular difference between image and model lines in the image */ 
float cont; /* confidence value for the match */ 

float dist; /* distance between the *match LINE and IMG_LINE */ 

float scale; /* ratio IMG_LINE->dmajor /MODEL_LINE->length */ 

SUFUCL MNALCINTy pes nes: 


PMA TCH) Pe: 


typedef struct line_type 


char name[3];: 
POINT pl, p2; /* the 2 endpoints for the linc’ 7/ 
struct line_type *next; /* ptr to the next IMG_LINE in the image */ 


/* Least Squares Fit mommenls: ------------------------+--+----- 4) 
long mQQ; /* Number of pixels */ 


double m10; /* Sum  */ 

double mQ1: /* Sum y */ 

double mit; 7* Suny x*y */ 

double m0: Sui se 

doublet]: /* Suny y */ 

double phi; /* Calculated normal orientation of IMG_LINE */ 
double dimajor: /* Length of major axis of equivallent ellipse */ 
double dminor; /* Length of minor axis of equivallent ellipse */ 


double rho; /* Ratio dminor/dmajor */ 
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/* Pattern Matching Information: --------------------------------- * / 
double angle_to_image_center; 


MATCHTYPE *matchlist; /* List of matches to LINE types */ 
MATCHTYPE *pm; /* present match being considered */ 
} IMG_LINE,; 
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APPENDIX B- EDGE EXTRACTION ROUTINES 


INPLEMENTATION: FINDEDGE.C 


[RR RATE ERNE ERK ARETE NENA EELS EE AIAN ED Ge ek 


/* FILENAME: findedge.c 
/* AUTHOR: Kevin Peterson 
/* DATE: 05 December 1991 
*K 


/*DESCRIPTION: An image gradient program incorporating edge-finding. 
‘ai Displays edge-gradient image and associated lines. 
ee 
/* This applicauon is designed for use on a Silicon- 
/* Graphics Iris (r) workstauon ullizing a .sgi 
/* or similar rgb formatted image. 
/* RGB values are of type LONG in the form AABBGGRR where: 
/* AA - alpha value, 0-255 
/* BB - blue component, 0-255 
/* GG - green component, 0-255 
/* RR - red component, 0-255 
a 
/* SiliconGraphics graphics library functions used within 
/* the display_bw_and_gradient_images(Q routine: 
/* qdevice(), winset(), c3f0, move20, draw2(), 
/* swapbuffers(), reshapeviewport(), winclose(), 
/* and lrectwriteQ. 
/* NPSIMAGE function routines borrowed courtesy of M.Zyda: 
/* read_sgi_rgbimage(), gel_emply_rgba_npsimage(), 
/* get_empty_rgb_npsimage(), and rgbalong_to_bwlong(). 
* 


fe Least Squares Fit method for line-finding from 

ee “Sonar Data Interpretaion for Autonomous Mobile Robots” 

ie by Y.Kanayama, T.Noguchi, & B.Haruman, 1990. 

[RE RR RE I ee I eae a a eee 
#include <gl.h> /* SiliconGraphics (r) graphic library */ 

#include <gl/image.h> /* SGI image structure library */ 

#include <device.h> /* Machine-dependent device library for keys and mouse-buttons */ 
#include <stdio.h> /* C standard i/o library */ 

#include <math.h> /* C math library for atan2() */ 


#include “imagce_lypes.h” /* Type definitions for NPSIMAGE, etc. */ 
#include “edge_types.h” /* Type definitions for EDGE, LINE, etc */ 
#include “npsimagesupport.h” /* Some NPSIMAGE funcuons */ 
#include “edgesupport.h” /* EDGE and IMG_LINE building functions */ 
#include “displaysupport.h” /* Graphics display functions */ 


#define THRESHOLD 5000000.0 /* for gradient magnitude threshold test */ 
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main(arge, argv) 
int argc; 
char *argv(]; 


NPSIMAGE *imgl, /* input file_name.rgb color image */ 
*img2, /* black&white image */ 
*img3; /* gradient image */ 


/* pointers to RGBA longs (“bitsptr’s) of respective NPSIMAGEs */ 
long *ptrl, *ptr2, *ptr3; 


double dx, dy, Th = THRESHOLD*THRESHOLD; 
register int i = Q, /* counter for pixels in input image */ 

z = (0); /* counter for pixels in gradient image */ 
EDGE “*reg; 


if(arge != 2) fatal(“usage: findedge filename\n”); 


/* Read in input rgb image */ 
img1 = read_sgi_rgbimage(argv[1}); 
ifamg] ==(NPSIMAGE *) NULL) 


fatal(“File %s is a NULL image.\n” img}->name); 


Xdim = img1l->xsize; /* else set global Xdim and ptrl */ 

ptrl = img1->imgdata.bitspu; 

printf{(“findedge:> %s xsize= %d ysize= %d pixels= %d\n”, 
img1->name,img1->xsize,img1->ysize, (img1l->xsize*img1->ysize)); 


/* Declare new NPSIMAGEs */ 
if((img1->type == RGBAWITHALPHA) Il (img1->type == RGBA)) 
{ 


img2 = get_emply_rgba_npsimage(Xdim,img1->ysize,img1->name); 
img3 = get_empty_rgba_npsimage((Xdim-2),img1->ysize-2,” findedge’”’); 


else 
fatal(““Unknown or c-mapped image type: %d.n”,img1->type); 


ptir2 = img2->imgdata. bitsptr; 
ptir3 = img3->imgdata.bitspw; 


/* The scan of an RGB image is from the bottom row -> up, 
traversing the rows left to right. */ 


/* In order for the Sobel operator to be calculated for a specific pixel, 
all eight surronding pixels must have an absolute (black & white) 
light intensity calculated. Function rgbalong_to_bwlong() performs 
this task. */ 


/* Due to the nature of the Sobel operator, the pixels in the top and 
bottom rows as well as pixels in the leftmost and nghtmost columns 
of the input image will not be calculated. In order to start cal- 
culating the Sobel operators, the first 2 rows of the input image 
must be converted to black & white light intensity values. */ 
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/* Calculate bw values for first 2 rows of inpul image. */ 
for(i=Q; 1<(2*Xdim)+2; ++1) 
{ 

rgbalong_to_bwlong(pul[1],&ptur2[1)); 


ford = Xdim + 1;1 < (Xdim*(img1->ysize-1))-1; ++i) 


/* Convert coloramg1]) to b/wUmg?2) for pixel on next row up and one 
pixel over to the right so that all eight neighbors of pixel 1 
have black & white light intensities. */ 


rgbalong_to_bwlong(ptrl [i+ Xdim+ 1],& ptr2[i+Xdim+1]); 
/* Ensure pixel 1 is not in leftmost ro rightmost column */ 
if(i%Xdim '=0) && G%Xdim != Xdim-1)) 


/* Calculate dx,dy via Sobel operator for pixel 1. */ 


dx = (-pur2(1+Xdim-1] + pu2[i+Xdim+1]] 
-(2 * ptr2[i-1])+ (2 * ptr2[i+1]) 
-pur2[i-Xdim-1) + ptr2[1-Xdim+1]); 


dy = ( pu2[1+Xdim-1) + (2*ptr2[1+Xdim]) + ptr2[i+Xdim+ 1] 
-pur2[1-Adim-1] - (2*ptr2{1-Xdim]) - pu2[1-Xdim+1)); 


i (dx*dx)+(dy*dy) > Th) 


pixel_membership(z,atan2(dy,dx)); 
sct_pircl_black(&ptr3[z]); 

else 
set_pivel_w hite(& ptr3[z]); 

++/./* Increment the pixel counter z for the gradicnt image. */ 


/* If piacl iis in the leftmost column, do check_active_edges(). */ 
else 10% Adim == Q) 


ChCchBichine sedecs(): 
}/endior ys; 
/* Check remaining EDGEs for lines: */ 


reg = Pasi_cdee_list_heid:; 
while reg |= ele, 


line_test(reg); 
ICO =1Ceeenck 


/* Write the lies list to file “lines aaa 327 
write_all_lines(argv[1],1mg3->xsize,img3->ysize); 
printf{(‘Number lines found in %s = %d\n”, argv[1],Linecount); 
/* Display the black& white and gradient images on the screen. */ 
display_bw_and_gradicnt_images(img2,img3,Line_list_head); 


print{(‘findedge %s...done An” argv[1]); 
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IMPLEMENTATION: FASTEDGE.C 


ee ee eee RS ER EERE TE EEE EES ERE A EEE 


/* FILENAME: fastedge.c 

/* AUTHOR: Kevin Peterson 

/* DATE: 06 January 1992 

/* DESCRIPTION: An image gradient program incorporating edge-finding. 
/* Displays edge-gradient image and associated lines. 

ok 


/* This application is designed for use on a Silicon- 

/* Graphics Ins (r) workstation uulizing a .sgi 

/* or similar rgb formatted image. 

/* RGB values are of type LONG in the form AABBGGRR where: 

/* AA - alpha value, 0-255 

/* BB - blue component, 0-255 

/* GG - grecn component, 0-255 

/* RR - red component, 0-255 

/* SiliconGraphics graphics library functions used within the display_bw_and_gradient_images() 
/* routine: qdeviccQ), winselt(), c3fQ, move2(), draw2(0, swapbuffers(), reshapeviewportQ), wincloseQ), 
/* and lrectwrite(. 

/* NPSIMAGE function routunes borrowed courtesy of M.Zyda: read_sgi_rgbimageQ), 

/* get_emply_rgba_npsimage(), get_empty_rgb_npsimage(), and rgbalong_to_bwlong(). 

/* Least Squares Fit method for line-finding from “Sonar Data Interpretaion for Autonomous Mobile 
/* Robots” by Y.Kanayama, T.Noguchi, & B.Hartman, 1990. 

i ee A AEST L ERAS ERE A SER ERE] 
#include <gl.h> /* SiliconGraphics (r) graphic library */ 

#include <gl/image.h> /* SGI image structure library */ 

#include <device.h> /* Machine-dependent device library for keys and mouse-buttons */ 
#include <stdio.h> /* C standard i/o library */ 

#include <math.h> /* C math library for atan2() */ 

#include “image_types.h” /* Type definitions for NPSIMAGE, etc. */ 

#include “edge_types.h” /* Type definitions for EDGE, LINE, etc */ 

#include “npsimagesupport.h” /* Some NPSIMAGE funcuons */ 

#include “‘edgesupport.h” /* EDGE and IMG_LINE building functions */ 

#include “displaysupport.h” /* Graphics display functions */ 


main(argc, argv) 
INL argc; 
char *argv{]; 


NPSIMAGE *img; /* input file_name.rgb color image */ 
long *ptr; /* pointer to bitspu of NPSIMAGE */ 
long grmask = Ox0000ff00; 
double dx, dy, Jh= THRESHOLD* THRESHOLD; 
register inti = 0, /* counter for pixels in input image */ 

z = 0; /* counter for pixels in gradient image */ 
EDGE “reg: 


if(argc '= 2) fatal(“usage: fastedge filename\n”); 

/* Read in input rgb image */ 

img = read_sgi_rgbimage(argv[1]); 

if(img ==(NPSIMAGE *) NULL) fatal(“File %s is a NULL image.\n” img->name); 

printf(“fastedge > %s xsize= %d ysize= %d pixels= %d\n", 
img->name,img->xsize,img->ysize, (img->xsize*img->ysize)); 

Line_list_head = fastlines(img); 


display_line_image(img,Line_list_head); 
printfC‘fastedge %s...done.\n” ,argv[1)); 
} 
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INIPLEMENTATION: VERTEDGE.C 


BF 2 ACR RR EE A A EA I EE eA A a 9 6 
/* FILENAME: vertedge.c 

/* AUTHOR: Kevin Peterson 

/* DATE: 30 Januameloo2 

Kk 


/* DESCRIPTION: Same a fastedge.c but only returns vertical edges from image. 
fey 


[REAR EO AA He EE EP a 


#include <gl.h> /* SiliconGraphies (r) graphic library */ 

#include <gl/image.h> /* SGI image structure library */ 

#include <device.h> /* Machine-dependent device library */ 
/* for keys and mouse-buttons */ 

#include <stdio.h> /* C standard i/o hbrary */ 

#include <math.h> /* C math library for atan20 */ 


#include “image_types.h” /* Type definitions for NPSIMAGE, etc. */ 
#include “edge types ty 7* Type deiimitions tor EDGE SEINE Vcc, 


#include “npsimagesupport.h” /* Some NPSIMAGE funcuons */ 

#include “edgesupport.h” /* EDGE and IMG_LINE building funcuons */ 
#include “‘vertsupport.h” /* Verucal EDGE and IMG_LINE supplement */ 
#include “displaysupport.h” /* Graphics display functions */ 


main(arge, argv) 

INL argc: 

char *argv[]; 

NPSIMAGE *imyg; /* input file_name.rgb color tmage */ 

i(arge {= 2) latalC usaue: vertedse tilenanien’ | 

/* Read in input rgb image */ 

Img = read_sgi_rgbimage(argv{1]); 

if(aimg ==(NPSIMAGE *) NULL) fatal(“File %s is a NULL image.An” img->name), 

print{(’vertedge> %s xsize= Kd ysize= %d pixels= %d\n’, 
img->name,iMy->Xsize,ung->ysize, (img->xs1ze*IMgZ->ysizc)); 

Line_list_head = vertiinesumy); 


display_line_image(imyg,Line_list_head); 


printi(*vertedge “s...done.An” argy{[ 1); 
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FILE: NPSIMAGESUPPORT.H 


ee ae A RR A A Oe A EK AK A OK / 
/* FILENAME: npsimagesupporth 
/* AUTHOR: Kevin Peterson 

feet: 29 January 1992 

* 
/*DESCRIPTION: Collection of basic npsimage funcuons. 
a NPSIMAGE *get_empty_rgb_npsimage (long xsize, long ysize, char name[]) 
ce NPSIMAGE *get_empty_rgba_npsimage (long xsize, long ysize, char name[]) 
ps NPSIMAGE *read_sgi_rgbimage (char filename[]) 
ie void write_sgi_rgbimage (char filename[], NPSIMAGE *img) 


a A RE TE ak Oe OR A OR OR Bet A RR ek RE | 


/* NPSIMAGE *get_empty_rgb_npsimage (long xsize, long ysize, char name[]) 
* 


/* The following function reads in an SG] RGB image as an NPSIMAGE. 
/* - coutesy of M. Zyda, Naval Postgraduate School 


—_—_— wo —_— 


NPSIMAGE *xct_cmpty_rgb_npsimage(xsize,ysize,name) 
long xsize, ysize; /* the size the image should be in pixels */ 
char name|]: /* name to attach to the image */ 

NPSIMAGE *img; /* ptr to an NPSIMAGE */ 

/* allocate an NPSIMAGE header */ 

img = (NPSIMAGE *)malloc(sizeof(NPSIM AGE)); 


/* set the type of NPSIMAGE to RGBA */ 
img->type = RGBA: 


/* record the widths and height */ 
Img->xsize = XSizec: 
IMg’->ySIZC = ysi/e; 


/* allocate space for the name of the image */ 
img->name = (char * )malloc(sulen(name)+1); 


/* copy the name into the string allocated */ 

sircpy (img->name,name); 

/* allocate the mcmory for thc bitmap of the image */ 
img->imgdala.bitspu = 

(long *)malloc(sizeol(lony) * img->xsize * img->ysizc); 


/* return a ptr to this empty image */ 
relurn(img ): 
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/" NPSIMAGE *get_empty_rgba_npsimage (long xsize, long ysize, char name[]) 
/* The following function reads in an SGI RGB image as an NPSIMAGE. 
ee s 
NPSIMAGE *vet_empty_rgba_npsimage(xsize,ysize,name) 
long xsize, ysize; /* the size the image should be in pixels */ 
char name{]; /* name to attach to the image */ 
NPSIMAGE *img; /* ptr to an NPSIMAGE */ 
/* allocate an NPSIMAGE header */ 
img = (NPSIMAGE *)malloc(sizeof(NPSIM AGE)); 


/* set the type of NPSIMAGE to RGBA */ 
ing->type = RGBAWITHALPHA; 

/* record the widths and height */ 

IMI E->XSiZe = NSI7e, 

NIE =S Size = Size. 

/* allocate space for the name of the image */ 
img->name = (char * malloc(strlen(name )+1); 


/* copy the name into the string allocated */ 
sucpy(img->name nume): 


/* allocate the memory for the biumap of the image */ 
Img->imydata.bitspur = dong *)malloc(sizeof(long) * img->xsize * img->ysizc); 


/* return a pu to this empty image */ 
return(img }: 


/* NPSIMAGE *read_sgi_rgbimage (char filename[]) 
* 


/* The following funcuion reads in an SG] RGB image as an NPSIMAGE. 
/* -coutesy of M. Zyda, Naval Postgraduate School 


NPSIMAGE *read_sgi_rgbimage(filename) 
char filename}; /* input filename */ 
register IMAGE *image; 
NPSIMAGE *img; /* pu to an NPSIMAGE structure */ 


register int x,y; /* temp indices for each line of 
data from the sgi image. */ 


long *ptu; /* temp pointer for each word of the 
NPSIMAGE long image. */ 


short rbuf[4096], /* temp arrays to hold scratch info for */ 
gbuf[4096], /* processing an sgi image */ 

bbul[4096], 

abul[4096}: 


/* open an sgi image */ 
feainacc=(open(ilename, © ))== NULL ) 


fprintf(stderr, read_sgi_rgbimage: can’t open input file Zs\n""/,filename); 
FroeueMCNPSIATAGE 77 NULL): 


if(amage->zsize<3) 


fprintf(stderr,” read_sgi_rgbimage: this is notan RGB image file\n”); 
Femur NPSINTAGE “)INUEL): 


} 
/* here we should allocate an NPSIMAGE */ 
amiare-->7Si7¢ == 3) 


/* just allocate an rgb image */ 
img = get_empty_rgb_npsimage(image->xsize,image->ysize, filename); 


ise 


—_— MM —~ 


/* get an image with alpha */ 
img = get_empty_rgba_npsimage(image->xsize,image->ysize filename); 


/* geta pointer to the NPSIMAGE longs */ 
pir = img->imgdata. bitsptr; 
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/* foneach row of the images: =. 
for(y=0; y < img->ysize; y=y+1) 


/* read a 1r0W Olneds 7) 
getrow(image,rbuf,y,0); 


/* read a row of greens */ 
getrow(image,gbuf,y, 1); 


/* read a row of blues */ 
getrow(image,bbuf,y,2): 


/* if we have an image with alpha, get it */ 
ifamg->type == RGBAWITHALPHA) 


gerrow(image,abul,y,3); 
/* we must now Step across the row and set each long integer 


of the NPSIMAGE format by combining the info from the sgi 
rows. 


ai 
for(xs0; X << 1MG-Sxsi7G x =x |) 
{ 
/* compute the RGBa long to plug in and plug it in */ 
if(img->type == RGBAWITHALPHA) 
“pur = rbul|\! + (gbul[x] << 8) + (bbuf[x] << 16) + (abul[x] << 24); 
esc 
/* RGBA iniage with alpha forced to Oxff */ 
*ou = rbullx| + @bullx] << 8) (bute Olea 
} 
fF Siep ihe pur tothe nextone 
pur++; 


7 sCaraskOr  / 


/* we have set all the bytes Of ihe image: 
Now retum the pu to the NPSIMAGE structure. 
a 


return(aimg): 
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/* void write_sgi_rgbimage (char filename{], NPSIMAGE *img) 
* 
/* The following funcuon wites an NPSIMAGE to a file. 
/* -coulesy of M. Zyda, Naval Postgraduate School 
void write_sgi_rgbimage (filename, img) 
char filename[]; /* output filename */ 


PSIMAGE *img; /* pu to an NPSIMAGE structure */ 


register IMAGE *image; /* a ptr to an sgi image structure */ 

register int \,y; /* temp indices for each line of data from the sgi image. */ 
long *pu; /* temp pointer for each word of the NPSIMAGE long image. */ 
long dimen; /* dimension of this mage */ 


short rbuf[4096], /* temp arrays to hold scratch info for */ 
gbuf[4096], /* processing an sgi image */ 

bbul[40906]. 

abuf[4096}: 


/* set the dimension and zsize of this image */ 
ifamg->lype == RGBAWITHALPHA) 


auineniaj-t: 
else 


/* RGBA image without alpha */ 
Ginen =. 3. 


/* open an sgi rgb image for writing */ 
image=lopen(fiicname, w” ,.RLE(]),dimen,img->xsize,img->ysize,dimen); 


/* get a pointer to the NPSIMAGE longs */ 
pir = 1mg->1mydata. bitsptr: 


/* for each row ol the image ... */ 
fory=0. 4 < inle-Sysize: y=y+1) 


/* we Must now step across the row and decode each long integer 
of the NPSIMAGE format into the 16 bit shorts sgi requires */ 


for(x=O0; x < IMg->Xs1z0; \=X+1) 
/* get the colors from the longs */ 
rbuf[x] = *pur & OxOOOOOOMT; 
gbul[x] = (*pu & OxOQ00ff00) >> 8; 
bbuf[x] = (*pur & OxO0ff0000) >> 16; 
abuf[x] = (*pu & OxffOOOO0Q) >> 24; 
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/* step the ptr to the next long */ 
pu++; 


/* write a row of reds */ 
putrow(image,rbuf,y,0); 


/* write a row of creens 7/ 
putrow(image,gbuf,y,1): 


/* write a row of blues */ 
putrow(image,bbuf,y,2): 


/* write a row Of alphus, if any */ 
ifamg->type == RGBAWITHALPHA) 
{ 


putrow(image,abul,y,3); 


/* we must close the outpul sgi image file */ 
iclosec(image): 
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FILE: EDGESUPPORT.H 


Dene ie cae ee EES EER EE AEE EERE REN AEE ® 


/* FILENAME: edgesupport.h 
/* AUTHOR: Kevin Peterson 
/* DATE; 29 January 1992 


*K 


/*DESCRIPTION: Collection of edge finding functions. 
* 


/* void fatal(char message) 

/* int gradient_angles_close(double r,double s) 

/* int close_to_negauve_pi(double phi) 

/* int horizontal(EDGE *r) 

/* EDGE *create_edge(long z,double phi) 

/* void add_pixel_to_edge (long z, double phi, EDGE *r) 
a DOL combine coses(EDGE *rl, EDGE *r2) 
fale PINE “create line (EDGE *r, double M20, double M11, double M02, 
/* double Dmajor, double Dminor, double Rho) 

Pv ole linestes! (EDGE *r) 

/* void check_actuuve_cdges () 

/* void rgbalong_to_bwlong (long rgbalong, long *bwlong) 
/* void pixel_membership (long z, double phi) 

/* void set_pixcl_white (long *rgbalong) 

/* void set_pixel_black (long *rgbalong) 

/* void write_all_lines (long x, long y) 

/* IMG_LINE *fastlines (NPSIMAGE *img) 


nn Na a ee ee EE Fe OE ON / 


/* Gradient Magnitude Threshold - for fastlinesiimg) */ 
#dcfine THRESHOLD 30000.0 


/* Gradient Angular Oricntauion */ 
FueimicneASVDELTA PHI0S /* maximum difference Gn radians) */ 


Constants {or function: vol line_test (EDGE *r) */ 

sOcinewIN er IX ELS PER LINE 60 /* mininvum pixels allowed for a IMG_LINE */ 
#define MIN_DMAJOR 20.0 /* minimum major axis Iength allowed */ 

#define MAX_RHO 0.1 /* maximum ratio (Rho=Dminor/Dmajor) */ 


#deline Pl 3.14159265 


/* --- Global variables ----------------------+-----------+--+-+---+---- =) 
long Xdim; /* width of input image (nr pixels) */ 
long Ydim; /* width of input image (nr pixels */ 


int Linecount = (0): /* counter for number of IMG_LINEs made */ 


poe oimlers &. Fe rescnt row EDGE list, Past row EDGE list, IMG_LINE list*/ 
BRUGES rresentcdvc_list_head = NULL, 

fienescntecdce list tail= NULL, 

Beacmeucenitst bend = NULL; 


iGwet Nie etne list. head= NULL; 
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/* void fatal (char message) 
Hi 


/* Prints error message and exits out of the program. 


fatal(message) 
char *messaye; 
{ 
fprintiGider, Fal eRRORS ); 
perror(message); 
eX 1h exiubvetcmune 7 


/* int gradicnt_angles_close (double r, double s) 

fs 

/* Retums 1 if gradient angle orientations of EDGEs r and s 
/* are within MAX _ DEET APH. 

/* Retums 0 otherwise. 


int gradient_angles_close(r,s) 
double r,s; 


if(@ < MAX_DELTA_PHI - PI) && (§ > 0.0)) r= Pl + PI +7; 
else 1106 < MAX _DELETAPHI~ Pieces O10) so = Flees, 


return(fabs(r - s)< MAX_DELTA_PHI): 


Kerr SiGe tm my a a ae aa oa a ROT saa e a i‘ a gk Gt ot ace ee a of 
/* int Close_to_negauye_pt (double phi) 
Ee 


/* Retums 1 if orentauion phi is within MAX_DELTA_PHI to -PI. 
/* Rétums 0 oihcmuisc: 


int close_to_negative_pi(phi) 
double phi: 


{ 
return(Pl + phi < MAX_DELTA_PHI); 


/ int honzonuiteporc) 

sh 

/* Retums 1] if orientation of EDGE e (r->avg_phi) is within 

/* MAX_DELTA_PHI/4 to PI/2 or -P1/2 (the normal orientations for 
/* a horizontal line). 

/* Retums O otherwise. 

int horizontal(¢) 


EDGE *e: 
{ 
double maxphi = MAX_DELTA_PHI] / 4.0; 


return((fabs(e->avg_phi) > (0.5*PI)-maxphi) && 
(fabs(e->avg_phiy < (0.5*Pl)+maxphi)); 
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/ 

/* EDGE *create_edge (long z, double phi) 

is 

/* Returns pointer to an newly instanuated EDGE with variable 
/* based upon the input pixcl z and orientation phi of pixel z. 

* 


EDGE “create -edge(z, phi} 
fom ze) zt pixellin image */ 
double phi; /* gradient orientation of pixel z */ 


GE * Tr; 
long x = z%(Xdim-2), /* (x,y) coordinates of pixel z in 2D image */ 
y = z/(Xdim-2); 


y allocate meinory for EDOE r */ 
f(r = (EDGE *)malloc(sizeof(EDGE))) == NULL) fatal(“create_edge: malloc\n”); } 


Peelseamialize liclds of EDGE r */ 
r->acuve = 0; 

feeirs) (pixel = 7: 

fe lAsiepIxcl— 7: 

eX = - 

fe xiax = xX; 

r->avg_ phi = phr: 

r->sum_phi = phi: 


emul = |: 
emi = x: 
emo! = y: 


r->m1] = x*y: 
fom = XX: 
ein 
fee xt = iN i: 


Heemescnmecadce lstehead == NULL) 
miesenlccdse list head =r: 
eerescnt cave list til '= NULL) 
rescniscdee [ist tal->ncvl = 1 
Presewiecadme list tail = 1: 


relurn(r): 
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/* void add_pixel_to_cdge (long z, double phi, EDGE *r) 
ies 
/ 


* Updates EDGE r with new pixel z and orientation phi of pixel z. 
pe * 


add_pixel_to_edge(z, phi, r) 

long 27 = Zui pix cl) 

double phi; /* gradient orientation of pixel z */ 
EDGE *1;/" EDGE (to add piste ic. 7 


long x =z%(Xdim-2), /* (x,y) coordinatcs of pixel z in 2D image */ 
y = z/(Xdim-2). 


if((r->avg_phi > 0.0) && close_lo_negative_pi(phi)) 

phi = PI + phi + Pl; 
else if((phi > 0.0) && closc_to_negative_pi(r->avg_phi)) 
{ 


r->ave_phi = Pl + r->uve_phi + PI; 
r->sum_phi = r->avg_phi * r->m00; 


/* update the new xmas for this row and last_pixel added to this EDGE */ 
r->xMax = Xj 
r->last_piacl = z: 


/* update the least squares fit moments */ 
++r->mO00; 
r->m10 += x 
r->mOl] += y 

Prem) =: 
r->m20 += A*N: 
remus +=). 


. 
$ 
e 

? 


/* recalculate the average gradient orientation for the EDGE */ 
r->sum_phi += phi; 
r->avg_ phi = r->sum_phi / r->m00; 
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(eeeer comemescaucs (EDGE *r], EDGE *r2) 
Ife 
/* Retums pointer to EDGE rl after appending all information 
POPE bGE 2 with rl. 
Has vse cen cases i aS Ee ROPES SG RRP a IO en Ge HON aE gg mae 
ER@OGE “combine cdecs(rl.r) 
Eb Gie "rie re, 
{ 


/* Special case where r1->first_pixel is changed: */ 
if(!}(r2->first_pixel > rl->first_pixel) fl (horizontal(rl) & & (rl->xmin < r2->xmin)))) 


rl->first_pixel = r2->first_pixcl; 
/* Modify last_pixel */ 
i(r2->last_pixcl > r1->last_pixcl) 


rl->last_piact = r2->last_pixel; 
lexis = fe->KININ: 


Pisce Wor zonliirs) Ac (r2->xmax > rl->xmax)) 


fee list pix) = 12->last_pixel: 


/* Special case: if gradient angles on opposite sides of +/- pi */ 
if((rl->avg_phi > 0.0) && close_to_negalive_pi(r2->avg_phi)) 
{ 


peeve pn = Pl + r2-suve phi Pl: 
2->sum_phi = r2->ax¢_ phi * r2->m00: 


else if((r2->ay g_phi> 0.0) && close_to_negative_pi(rl->avg_phi)) 


— 


eed erp — Pl + rl->uve phi + Pl: 
rl->sum_phi = rl->avg_phi * r1->m00: 


/* combine least squares fit moments */ 
rl->m0OO += r2->m00: 
rl->m10 += r2->m10: 
rl->m0Q1] += r2->mQ1: 
roe) 1 += bo->in | |: 
feem20 += 52->im20: 
rl->mQ2 += r2->m0?. 


/* combine and recalculate average gradient orientation for the EDGE */ 
rl->sum_phi += r2->sum_phi; | 
rl->avg_phi = rl->sum_phi /rl->m00; 


return(rl); /* return EDGE rl with combined information of rl & r2 */ 


9] 


weeeeewen eee eee ewe eee wee we Se wee ee ee eee ew wwe we ew we wwe ee eese ees ee eee ee ere eses 


/* IMG_LINE *create_line (EDGE *r, double M20, double M11, double M02, 
/* double Dmajor, double Dminor, double Rho) 

ie 

/* Returns pointer to newly instanuuated IMG_LINE with variables set 

/* according to moments described by EDGE r, secondary moments 

/* M20, M11, and MO2, axis Iengths Dmajor, Dminor, and ratio of 

/* axis lengths Rho. 


Vie ae eee eS a I ee aE ae aa 
IMG_LINE *create_line(r,M20,M11,M02,Dmajor,Dminor,Rho) 
EDGES? 
double M20,M11,M02,Dmajor,Dminor,Rho; 


IMG_LINE *1; 


/* r->first_pixcl mapped onto the IMG_LINE will be endpoint pl] 
r->last_pixel mapped onto the IMG_LINE will be endpoint p2 */ 


long X1 = r->first_pixcl%(Adim-2), 
yl =r->first_pixel/(Adim-2), 
x2 =r->last_piac]l&(Xdim-2), 
Ve =f laste pinclencun-2 


/* Calculate the normal orientauon of the IMG_LINE by atan2() funcuon. */ 
double Pht =auin227 Mi TiN 212072 0: 


/* Deltal and della2 are the offsets used to calculate the endpoints 
for the IMG_LINE segment based upon values x1,y1 and x2,y2. */ 


deltal = (r->m10/r->m00 - (double)al)*teos(Phi) + 
(r->mO1/r->m00 - (double)y1)*fsin(Ph1), 

delwa2 = (r->m10/r->m00 - (double)x2)* fcos(Phi) + 
(r->mO1/r->m00 - (doublejy2)*fsin(Phi); 


/* Phi = atan2(-2%M11,N102-M20)/2.0) always returns positive result to Phi. 
Therefore, negative_phi = (r->avg_phi < (0.0) is necessary, */ 
int negative_phi = (r->avg_phi < 0.0); 


/* Allocate memory for IMG_LINE 1, */ 
iff = UMG_LINE *)malloc(sizeofIMG_LINE}))) == NULL) fatale crearesiine malloc ms: 


/* Calculate x,y coordinates for endpoints p1 and p2. */ 


l->p1.x = (double)x 1 + deltal *fcos(Phi); 
l->pl.y = (double)y1 + deltal*fsin(Phi): 
l->p2.x = (double)x2 + delta2*fcos(Phi); 
l->p2.y = (double)y2 + delta2*fsin(Phi); 


/* Copy least squares It moments. */ 


l->m00O = r->m00; 
]->m10 = r->m10; 
l->mQ1 = r->m01; 
l->m1lil=r->m11; 
l->m20 = r->m20; 
Eem02 = 7->m02- 


/* Phi ts positive, but -pi < r->avg_phi < pi. */ 
if(negative_phi) |->phi = -Phi:; 

else 1->phi = Phi: 

Voge pdate res! 01 IMG LIEINE Values. */ 
next = NUL, 

1->dmajor = Dmajor: 

1->dminor = Dminor: 

l->rho = Rho; 

1->matchlist = NULL: 

E=pny= NULL: 


++Linecount; /* Increment global variable, Linecount. */ 
sucpy(1->name,” “); 

sprint{(->name, “%d”", Linecount): 

feturntl); /* Return IMG_LINE |. */ 
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/ 
/* void lineéatcsm( EDGE. 1 
i 


/* Determines if EDGE r meets three requirements to be a IMG_LINE: 
/* (1) The number of pixels in EDGE r (r->mQ0Q) be greater than MIN_PIXELS_PER_LINE. 
/* (2) The ratio (Rho) of the length of major and minor axes of the 
/* EDGE be less than MAX_RHO. 
/* (3) The length of the major axis (Dmajor) be greater than MIN_DMAJOR, the minimum 
/* IMG_LINE length allowed. 
/* If all three conditions are met, anew IMG_LINE type Is created and 
/* appended to the Linc_list in order of significance (in this case Dmajor). 
ir ee ee ee eT ee OBE or ne a a | 
line_test(r) 
EDGE. 


IMG LINE ET *inscrtapre= Emme alisteheda: 
double M20,M11,M02,Ma,Mb,Mmajor,Mminor,Dmajor,Dminor,Rho; 


/* First test -- A]MG_LINE must have a required minimun number of pixels. */ 
if(r->m00> MIN PIX ERSSPERS@IEINEy 

/* Calculate secondary moments by least squares fit. */ 

M20 = r->m20 - ((r->m10*r->m10)/r->m00); 

M11 =r->m11 - ((r->m107r->m0Q1)/r->m00); 

MOQ2 = r->m02 - ((r->m01*r->mO1)/r->m00); 


/* Calculate major and minor axis lengths, Dmajor and Diminor, */ 
Ma = (M20+M02)/2.0; 

Mb = sqri( ((M02-M20)* (M02-M20)/4.0) + (M11*M11) ); 
Mmajor = Ma - Mb; 

Mimunor = Ma + Mb; 

Dmajor = +.0*sqri(Mmiunor/r->m0Q); 

Diminor = 4.0* sqruMimayor/r->m0Q):; 


CUCU TAO Oe 
Rho = Diminor/Dmayjor: 


/* Second & Third tests -- Ratio Rho must represent not a blob. 
-- IMG_LINE must be at least a certain length. *,; 

if((Rho < MAX_RHO) && (Dmajor > MIN_DMAJOR)) 

{ 


/* The EDGE passed the three requirmemis to be a line, */ 
1 = create_line(r,M20,M11,M02,Dmajor,Dminor,Rho); 


/* Add new IMG_LINE to IMG_LINE list in order by IMG_LINE length, dmajor. */ 
if(Line ist head ==NUEE) Pine slistahead =F 
else if(l->dmajor > Line_hst_head->dmayjor) 


{ 
l->next = Line_list_head; 
Pines stance id=). 

else 


while(Ginsert_pt->next != NULL) & & (J->dmajor < insert_pt->next->dmayjor)) 
inserl_pl = insert_pl->next; 

l->nexl = Inser_pt->next; 

iscM ope leat — 


} /* end if second and third tests */ 
1? ene 1 tirstiest 7 
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/* void check_active_edges (0) 

hs 

/* This function, performed at the start of scanning for each row, 
/* cycles through the EDGEs in the Present_edge_list to see if: 
/* (1) Adajacent EDGEs on the same row have gradient oricntauions 
/* that are close and therefore can be combined into one 

Pee DOE. 
/* (2) EDGEs from the Present_edge_list are adjacent to 

/* EDGEs from the past row’s Previous_edge_list and their 

/* gradient orientations are close so that they may be 

/* combined into one EDGE. 
/* The second method for determining when to combine_edges() 
/* provides the means for constructing EDGEs across rows. 

2k 


/* AILEDGEs not combined with a EDGE from the past row are 
/* tested by linc_test() which examines the EDGE for satisfying 
/* the IMG_LINE requirements and appends the IMG_LINE to the Line_list. 


oa mae apart as ee mt SOIT Se PPE IE ie, Dect ne fe one oh a 8 0 a | 
check_active_edges() 


POE pres resem cdec list head, /* the EDGEs found by the scan of this row */ 
*nast = Past_edge_list_head, /* the EDGEs found during the scan of the previous row */ 
*temp: /* a temporary pointer used for frecing memory */ 


Int conunuc_loop; /* a boolean integer */ 


/* Loop through all EDGEs in the Present_edge_list (all EDGEs found 
during the scan of the present row. */ 


while(pres '= NULL) 


f ook lorward to the ncxt EDGES on the prescnt row: to sce 
any should be combined with this EDGE. If so, combine_e 
ancemec ithe second EDGE. */ 


if 
dges() 


while((pres->next != NULL) & & (pres->xmax+1 == pres->next->xmin) && 
vradicnt_angles_close(pres->avg_phi,pres->next->avg_phi)) 


( 
pres = combine_edges(pres,pres->next); 
LcMp = pres->nexh; 
pres->NneAxl = pres->nexl->nexl; 
free(lemp): 


/* Set conunue_loop boolean to wue and loop through the EDGEs of 
found during the scan of the previous row of pixels. */ 
coniimuceloop = |: 


while((past '= NULL) & & (pres->xmax > past->xmin-2) & & (contunue_loop)) 


( 


/* If the two EDGEs are adjacent and their gradient orientations 
are Close, then the information of the past EDGE must be in- 
cluded with the present EDGE and the EDGE should be marked 
that it is sull “acuve™. */ 


if{((pres->amin < past->xmax+2) & & gradient_angles_close(pasi->avg_phi,pres->avg_phi)) 


pres = combine_edges(pres, past); 
past->active = 1; 


/* If the past EDGE was not appended to a present EDGE, (1.e. 
not “acuve’) then no more pixels may be added to it and it 
shall be tested if it satisfies the conditions for an IMG_LINE. */ 


if((past->next '= NULL) & & (pres->xmax > past->next->xmin-2)) 


il(!past->active) line_test(past); 
Dust = past->next} 


/* If the past EDGE did not meet any of the above two conditions, 
then exit this loop and continue evaluation with the next 
EDGE of the presenvrow. 7 


else continue_loop = 0; 


DIGS = pies- sles): 


/* Continue line_test() and tree() all EDGEs in the past row. */ 
while(past '= NULL) 

(CmMp = past: 

I('past->ucuve) line_lest(pust); 

Past = pusl-Sncx 

free(temp): 


/* Finally: Past row <- Present row, Present row <- empty. */ 
Past_edge_list_hcad = Prostnimedre slistshead 
Presenicdecalistahe d= ee 

Present cde aise il =e ee. 
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/* void rgbalong_to_bwlony (long rgbalong, long *bwlong) 

hi 

/* Converts color to black/white for rgba formatted pixels. 

/* The weights assigned for cach color are television standards. 
/* This function courtesy of M. Zyda. 


rgbalong_to_bwlong(rgbalong, bwlong) 
long rgbalong; /* input color rgbalong */ 
long *bwlong; /* output b/w rgbalong */ 

{ 

unsigned char red, green, blue, alpha; 

unsigned bw; 


/* Use bit masks to get RGB and alpha values from input rgbalong. */ 
red = rgbalong & OxOOOOOOIT; 

green = (rgbalong & OxO000IT00) >> 8; 

blue = (rgbalong & OxOOff{O00Q) >> 16; 

alpha = (rgbalong & OxffOOQ0000) >> 24; 


/* Calculate the black& white intesity using NTSC standard. 
intensity = 0.299(red) + 0.587(green) + 0.114(bluc) */ 
bw = (0).299* red)+(0.587* green)+(0.114* blue): 


/* Save the black&white intensity in bwlong. */ 
*bwlong = (alpha<<24)l(bu <<16)l(bw<<8)ibw; 


/* void pixel_membership (long z, double phi) 
* 


/* For a given row while scanning the black& white image, (scan of row> ....yyyyy....PITTrrr4rrz 
Poy — pivcls ingluded in previous EDGE ol saime row, 
foo = pincls included in EDGE Ff, 
/* z= pixcl z being tested for inclusion with EDGE r) 
ia 
/* two requirements must be sausfied for a pixel (z) with the pixels of a given EDGE: 
/* (1) pixels must be adjacent (1.c. the last_pixel of the EDGE + 1] must equal pixcl z), 
/* (2) the gradient angle onentatrons of the last_pixel and z must be “‘close”. Otherwise, pixel z 
/* can not be included with EDGE r and must be considered the first_pixel ofa new EDGE for this row. 
Yh gga en ee, CO fT ES eT =) 
pixel_membership(z, phi) 
long z: /* the zth prvel of the gradicnt mage */ 
double phi: 


BOGE *r: 

if((Present_cdgc_list_tail ‘= NULL) && 
(Present_edge_list_tail->last_pixel+1 == z) && 
gradicnt_angles_close(Present_edge_list_tail->avg_phi,phi)) 


add_pidrcl_to_edge(z,phi,Present_cdge_list_tail); 


else /* create new EDGE and add to end of Present I1st */ 


{ 


f= create. edsc(z phi): 


Sy 


/* void set_pixel_white (long *rgbalong) 

js 

/* Sets the specified long integer pointed to by rgbalong to be 
/* white by setting all RGB bits to ff (255 dec -> max intensity). 


set_pixel_white(rgbalong) 
long *rgbalong; 


ny 


/* void set_pivxel_black (long *rgbalong) 

hg 

/* Sets the specified long integer pointed to by rgbalong to be 
/* black by setting all RGB bits to 00 (O dec -> min intensity). 


na dceeedenedn tinged. chs foe eee te * / 
set_pixel_black(rybalony) 
long *rgbalong; 
*rebalong = OxtfOOOQOO0: 
[ea gree es ce ii 57. cr, amie Cane © aes pe 
/* void write_all_lines (lony x, long y) 
hx 
PO TMCW.OULUL Mile “HumMcsle ns 
fh Pee) eh ee Ree ea RPA eee ey ork Bee se nn ee th ay 


write_all_linesia,y) 
Jong x,y: /* the x,) dimensions of the image */ 


IMGEEINE 4) bine csi nee 
FILE *lites sie: 


lines tile =fopen’ lmestext 5 4), 


fprinti(lines_ file," d “d\nwn".x,y); 

white =NUB) 

{ 
fprintf(lines_tile,’% .2t %.2hn%.2f %.2&\n%.4Mn”, 
J->p1.x,l->pl.y l->p2.x,I->p2.y,l->phi): 
l= |->nAext: 


} 
fclose(lines_file): 


prinui(” lines found in image written to: ‘lines.text’\n"); 
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/ 
/* IMG_LINE *fastlines(NPSIMAGE) 
i 


wane wte were eee ee ee we Ee ee eR EB RP ewe Eee Ee EB BP ee eR KP ewe KF ww ww wm ew www eww eee e ee ee ee 


/* Retums a pointer to the head of a linked list of vertical IMG_LINEs lound in an NPSIMAGE. 
* =} 


IMG_LINE *fastlinesiimg) 
NPSIMAGE *img; /* input file_name.rgb color image */ 


{ 

long *ptr; /* pointer to bitspur of NPSIMAGE */ 

long grmask = OxOOOOff00; 

doupledx.dy, 1h= JHRESHOLD* THRESHOLD; 

register inti = QO, /* counter for pixels in input image */ 
z= 0; /* counter for pixcls in gradient image */ 

BOGE ree: 


Xdim = img->xsize; 

Ydim = img->ysize; 

pu = img->imgdata.bitspu; 
Einecount = 0: 


/* The scan of an RGB image is from the bottom row -> up, traversing the rows Ieft to right. */ 
fora = Xdim + 1,1 < (Xdim*(img->ysize-1))-1; ++1) 


sip cliis im the lelimoshcolummedo check active_cdecs(). */ 
1% Adim == 0) check_active_edges(); 


/* Ensure pixel tis notin leftmost or rightmost column */ 
else lay Adiny = Xdime)) 


/* Calculate da,dy via Sobe! operator for pixel 1. */ 


dx = (-(puli+Adim-1]]&grmask) + (pirti+ Xdim+1]& grmask) 
-(2 * (purfi-1J&grmask))+ (2 * (ptr[i+1]& grmask)) 
-(pur[i- Xdimi- 1 ]&ermask) + (pu[i-Xdim+1]&grmask)); 


dy = ¢ (pur[i+ Xdim-1]& grmask) + (2*(pu[i+Xdim]&grmask)) 
+ (pu[it+Xdim+]J& grmask) - (pu[i-Xdim-1]& grmask) 
- (2*(purli-Xdim]& grmask)) - (ptr[i-Xdim+ 1]& grmask)); 


if((da*dx )+(dy*dy ) > Th) pixel_membership(z,atan2(dy,dx)): 
+47: 
iaeendlor ) */ 
faseheck remaimmys EDGES for lines: */ 
reg = Past_cdge_list_head: 
while(reg !'= NULL) 
line_test(reg): 
reg = reg->nerl 


eave the lines list 16 file “lines.text’. */ 
write_all_lines(img->xsiz¢,img->ysize); 


prinu(’ Number lines found in %s = %d\n” ;mg->name,Linecount); 


fevmmieine list head): 
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ae 


FILE: DISPLAYSUPPORT.H 


[PRR OR EA AR ER A a ee 
/* FILENAME: displaysupport.h 

/* AUTHOR: Kevin Peterson 

/* DAVE? 3 Manuarysle?2 

DESCRIPTION: Collection of display functions. 


/* void display_bw_and_gradient_images (NPSIMAGE *img]1, NPSIMAGE *img2, 
ie IMGSEINE |) 
/* void draw_red_lines (IMG_LINE *]) 
i void display_loop (NPSIMAGE *img]1, long winid], 
NPSIMAGE *imgz2, long winid2) 
i void display_linc_image (NPSIMAGE *img, IMG_LINE *1) 
/* void display_line_loop (NPSIMAGE *img, IMG_LINE *1, long winid) 
Jigs 


/* These tuncuons make calls to following SiliconGraphies routines: 
/* prefsizeQ), winopend), winset(), wincloseQ, RGBmode(), 

/* singlebufferQ, gconfig(, qdevice(), c3fQ, move2(), draw2(), 

/* swapbulffersQ, Irectwrite(), reshapeviewportQ, and clear(). 


[RARE RA AHR ES Hee eA A eee EH A ek 


{= Bie ei te vee at ee SS oe ere re an eg ee OC * 
/* void display_bw_and_gradient_images (NPSIMAGE *img]1, NPSIMAGE *img2, 
ie IMG_LINE *]) 


/* Displays NPSIMAGE img] and img? on SiliconGraphics’ Iris workstauon. 
/* Each window will be displayed within red outline when operator depresses left mouse button. 
*h 


display_bw_and_gradicnt_images(img],img2,]) 
NPSIMAGE *imgl, /* input b/w image */ 
*img2; /* gradient image */ 
LNIGHEN NE | 
{ 


long winid], winid2: /* silicon graphics window 1d’s */ 


prefsize(img1->asize,mg]->ysize); /* preferred size for window */ 
winid] = winopen(img 1->namce); /* open the window */ 
RGBmode(): /* set RGBmode, singlebuffer, and */ 

singlebuller(); /* configure the window */ 

gceonfig(); 

prefsize(img2->xsize,1ng2->ysize); /* preferred size for window */ 
winid2 = winopen(img2->name); /* open the window */ 
RGBmode(): /* set RGBmode, singlebuffer, and */ 

singlebulfer(): /* configure the window */ 

geonfig(: 


/* iniuualize_controls * / 
qdevicec(REDRAW); 
qdevice(LEFTMOUSE); 
qdeviee(MIDDLEMOUSE): 
qdevieec(RIGHTMOUSE): 
Qde vice Seite 1): 


display_loopumeg] ,winid] ,img2,winid2,]); 


freeamg1]->imgdata.bitspu); /* delete the bitmap for the image */ 
freeame 1); * delete the NPSIMNAGE sineire 7/ 
winclose(winid]); /* close the window */ 

freeaamg2->i1mgdata. bitspu); 

irec(ime2); 

winclose(winid?): 
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draw_red_lines(]) 
ENIGELINE *1: 
{ 
Slane tioalred!5] = { 1.0,0.0,0.0): /* seb red */ 
eoled): 
wmeww=NULE) 
{ 
move2(l->p1.a,]->p1.y):; 
draw2(l->p2.x,]->p2.y): 
= We Snext: 


swapbulfers(): 


/* void draw_white_lines(IMG_LINE *1) 
je 
(rans white lines Over an iniave. 
draw_white_lines(]) 

MG LINE *1: 


{ 

Se oat winielo | =) WealbOo OU). /" eb white */ 
esiaw htc): 

wime(=NULL 

{ 


move2(l->p1.A.1->p].y): 
draw2(1->p2.,1->p2.y }: 
[= Teese: 


swapbulfers(): 


e@eewereewreonwmeoeceewereceee eee eee ece-—-e +--+ ee eee ee ee eee ee ee ewe ewe ew ew ww ew ee ew ew 


draw_black_linest]) 


IMGSEINE 71; 


SE iedoar olen | o) = |0.00.0),0.0}: /* rab black */ 


esitblack): 
mnie V=NULL) 


move2(->p).x!->pl.y): 
draw deen le policy): 
l= ence 


swapbulfers(): 
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Oe ee ee a eee eee eee 


avo draw black hnescihiG LINE *1) 
bs 


sai 


=} 


/* void display_loop (NPSIMAGE *img]1, long winid], 
/* NPSIMAGE *img2, long winid2) 
fe 
/* Continuously displays the white& white (img]) and gradient (img?) 
/* images within their respective windows (winid1] and winid?2). 
/* - Depress left mouse button when cursor arrow is in uule 
/* bar to see Jines initially over the image. 
/* - Depress middle mouse bulton when cursor 1s in image to 
Po isee lines ons 
/* - Depress middle mouse button when cursor is in utle bar 
/* “drag” window and redisplay image with lines. 
/* - Depress right or Ieft mouse buttons when Cursor is i 
/* either image to kill windows. 
tbs ee oi oS ee a eae een ee a 
display_loopamg1,winid] ,img2,winid?2,1) 
NPSIMAGE *img], /* input b/w image */ 
*imp2; /* gradiént image */ 
long winid], winid2; /* window id’s for images */ 
[MGR NE 


stauic float white|3)= (10, Osco rebiniiie 
short value: /* value muunecd Inonimihe evento, 


(display the images once 7 

winsel(winid] ); 

lrectwrite(O,0,1mg1l->asize - 1,img]->ysize - 1,img1->imgdata.bitsptr); 
winset(winid2): 

Irectwrite(O,0,1mg2->asize - 1,img2->ysize - 1,img2->imgdala.bitsptr); 


/* Joop unul a niouse button ts pressed */ 
chile Ree 


switch qread(& value) 


Cusc REDRAW: 
winsel((long )value): 
reshupe viewport): 
if(value == winid]) Irectwrite(O,0,1mgl->asize - 1,img]->ysize - 1, img]->imgdata.bitsptr); 
(value == wintd?) Irectwrite(O,0,1mg2->xsize - 1,img2->ysize - 1, img2->imgdata.bitsptr); 
draw _red_lines(1): 
break; 

Cie PERT NOUSE: 

Ce VIDDEE MOUSE: 
iG@alue == 0) | 

c3f(white): 

Clean); 

draw_red_lines(]): 
Dicak: 

Cise RIG IO SE: 
i@alue == Ui retin: 
break; 

CcSGESCKEN. 
exit(O); 

default: 
break; 

\ Pend Sein 7/ 

} /* end while */ 


display_line_image(img,]) 
NPSIMAGE *img; 
PIG LINE +1; 


long winid; 


prefsize(img->xsize,img->ysize); /* preferred size for window */ 
winid = winopen(img->name); /* open the window */ 
RGBmode(); /* set RGBmode, singlebuffer, and */ 
singlebufferQ; /* configure the window */ 

gconfig(); 


/* initialize_controls */ 
qdevice(REDRAW); 

Ode vice LEP IMOUSE): 
qdevice(MIDDLEMOUSE); 
qdevice(RIGHTMOUSE): 
Magevice (ESCKE Y ): 


display_hine_loopGimg,l,winid): 


freeamg->imgdata.bitspur); /* delete the biumap for the image */ 
freeimg); /* deletc the NPSIMAGE structure */ 
winclose(winid); /* close the window */ 


Ce ee ee ee ee eS eee ee eee ll 


www ew ew wa a ee ee ee ew ow me oe ee ee ee ee eee ee eee eee ee eee ee ee ee ee ee 


display, line —loopumg. ]winid) 
NPSIMAGE *img: 
NGeIEINE * 1; 
long wind: 


Pele noat wintel 3) = (1,051,0:1,0); * teb white */ 
Sontvalue.,* value rcturned trom theevent queue */ 


/* display the images once */ 
winsel(winid): 
Irectwritc(O,O,img->asize - | amg->ysize - 1,1mg->imgdata. bitsptr); 


/* loop until a Mouse bulton is pressed */ 
while(TRUE) 


switch(qread(& value) 


case REDRAW: 
winsel((long) value); 
reshapeviewporl(); 
if(value == winid) Irectwrite(0,0,img->xsize - 1,img->ysize - 1, img->imgdata.bitsptr); 
break; 
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Case ERI MOUSE. 
/* draw white lines over input image */ 
if(value == Q) draw_white_lines(]); 
break; 


case MID DEEN CU SE. 
ii(value == 0) 
{ 
c3f(white): 
clear(); 
draw_black_lines(1): 
J 
break; 


case RIGHTMOUSE: 
iftvalue == 0) roti 
break; 


CaS@ EO GCKET. 
Cx One 


default: 
break; 


ly Onda ie 


| PP enidewiiiie 37 
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FILE: VERSUPPORT.H 


Se a FE A A Ka ACE Re RK ER Re RK 


/* FILENAME: vertsupport.h 
/* AUTHOR: Kevin Peterson 
/* DATE: 29 January 1992 

xk 


/*DESCRIPTION: Supplementary vertical edge finding functions. 
Aint vertical(E DGE) 

/* void vertuical_line_tesuEDGE) 

/* void check_active_vertical_edges() 

/* IMG_LINE *verthines(NPSIMAGE) 


a re EE EE EE NE A EE A EE 


#define VERT_CONSTRAINT 0.0349 /* radians (= 2.0 degrees) */ 


Pome verical(EDGE) 

jee 

/* Retums 1 if orientauion of EDGE r (r->avg_phi) 1s within 

/* MAX_DELTA_PHI/4 of 0, PI, or -PI (the normal orientations for 
fy Veriical line). 

/* Retums O otherwise. 


mt vertical(r) 
OG ET: 
retum ((fabs(r->avg_phi) < VERT_CONSTRAINT) Il 
(fabs(r->ave_phi) > P] - VERT_CONSTRAINT)): 
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/* vord verucal ine lesiE GE) 


ie 
/* Same as line 1eésitr), bul also ensires tal EDGE Fis venie.r 
He ee ene RE Pe AT), erry om eRe RR Etat 7 
vertical_line_tcst(r) 

EDGES t. 


( 
IMG_LINE #1], *insen p= Eineslistencade 
double M20,M11,M02,Ma,Mb,Mmajor,Mminor,Dmajor,Dminor,Rho; 


/* First test -- A IMG_LINE must have a required minimun number of pixels. */ 
if(->m00 > MIN_ PIXELS PER EINE) @oetveniealiry) 
{ 


/* Calculate secondary moments by least squares fit. */ 
M20 = r->m20 - ((f->m10*r->m10)/r->m00); 
M11] =r->m11 - ((r->m10*r->m01)/r->m00); 
M02 = r->m02 - ((r->m01*r->m01)/r->m00); 


/* Calculate major and minor axis lengths, Dmajor and Dminor. */ 
Ma = (M204+M02)/2.0; 

Mb = sqri( (M02-M20)*(M02-M20)/43.0) + (M11*M11) ); 
Mmayor = Ma - Mb; 

Mmuinor = Ma + Mb: 

Dmajor = 4.0* sgrt(Mmunor/r->m00Q); 

Diminor = 4.0* syrtMimajor/r->m00): 


FPoCaleninierto oe 
Rho = Dminor/Dmayjor: 


/* Second & Third tests -- Ratio Rho must represent a line, not a blob. 
-- IMG_LINE mus! bDeaiicast a cenam lenginn: 7 
if((Rho < MAX RHO) && (Dmajor > MIN_DMAJOR)) 


/* The EDGE passed the four requirments to be a vertical line. */ 
l= create_line(r,M120,M111,M02,Dmajor, Dminor,Rho): 


/* Add new ING_EINE to IMG_LINE list im order by 
IMG CLINE lengthier 7 

deme lst head== NPE) ermesiistanesci= 

else #U->dmajor > Line_list_head->dmayor) 


. Snes = Vineslist eid: 
Lei alist 

cise 

{ 


winile(insenl_pi-snext = NUL Ak 
(]->dimajor < inserl_pt->next->dmayjor)) 


IMscrl_Pl = IMserl_pt->nexl 
l->nexl = Msert_pt->nexl; 
WSC ene t=]: 


\ /* endal-secondsind third tessa, 


j7* end i iisiies 7 
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/* void check_active_vertical_edges() 


i 
/* Same as check_active_edges(), but for verucal EDGEs only. 
‘ea mpeeyemsaieaeey ne es a psy ye ny Oe NS wt op I a a he ag Ns BN A el ee 


Checksactive verlical_edges(} 


PioGr pies — resent cu gee isi nead, 

/* the EDGEs found by the scan of this row */ 

*past = Past_edge_list_head, 

/* the EDGEs found during the scan of the previous row */ 
*temp; /* a temporary pointer used for frecing memory */ 


int continue_loop; /* a boolean integer */ 


/* Loop through all EDGEs in the Present_edge_list (all EDGEs 
found during the scun of the present row. */ 


while(pres != NULL) 
/* Look forward to the neat EDGEs on the prescnt row to sec if 


any should be combined with this EDGE. If so, combine_edges() 
ondainrce@ the second EDGE, */ 


while((pres->neat != NULL) & & (pres->xmaatl == pres->next->xmin) && 
gradicnt_angles_ close(pres->avg_phi,pres->next->avg_phi)) 


pres = combine_cdyges(pres,pres->nexl); 
(GMD = pres-SieN: 

pres->nexl = pres->nexl->next: 
ection): 


/* Set continue_loop boolean to true and loop through the EDGEs of 
found during the sean of the previous row of pixcls. */ 


contunue_loop = |: 
while((past '= NULL) && (pres->xmax > past->xmin-2) && (continue_loop)) 


/* If the two EDGEs are adjacent and their gradient orientations 
are Close, then the information of the past EDGE must be in- 
cluded with the present EDGE and the EDGE should be marked 
a ielsstill ace. */ 


if((pres->Aamin < past->xAmax+2) && 
gradicnt_angles_close(pasl->avg_phi,pres->avg_phi)) 
pres = combine_edges(pres, past); 
Pusl->acutive = 1: 


/* If the past EDGE was not appended to a present EDGE, (1.e. 

not “acuuve”’) then no more pixels may be added to it and It 

shall be tested if it satisfies the conditions for an IMG_LINE. */ 
i((past->neat '= NULL) && (pres->xmax > past->next->xmin-2)) 


if({past->active) veruical_line_test(past); 
pasl = past->neal; 
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/* Wf the past EDGE did not meet any of the above two conditions, 
then exit this loop and contnue evaluation with the next 
EDGE of the presenta w= 


else continue: loop =: 


pres = pres->next. 


/* Conunue verucal_line_test() and freeQ) all EDGEs 1n the past row. */ 
while (past.= NUE) 
{ 


lemp = past; 

if(!{past->active) vertical_line_test(past); 
past = Past->Next: 

free(iemp); 


/* Finally: Past row <= Presentomy: tesenhiom =< cliph ee 
Past_cdee_hHst_nead = Piesenteedzeanstancad, 
Present_edge_list_head = NULL: 

Present_cdvesheratiil = ieee: 
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/* IMG_LINE *vertlines(NPSIMAGE) 

hs 

/* Retums a pointer to the head of a linked list of verucal 
/* IMG_LINEs found in an NPSIMAGE. 


IMG_LINE *vertlines(img) 
NPSIMAGE *img; /* input file_name.rgb color image */ 


{ 

long *pu; /* pointer to bitspu of NPSIMAGE */ 

long grmask = OxOOOOff00; 

double'dx, dy, [h= THRESHOLD*THRESH@LD; 
register inti = O, /* counter for pixels in input image */ 
z= 0; /* counter for pixcls in gradient image */ 

EDGE rer; 


Xdim = img->,s12z¢; 

Ydim = img->ysize; 

pir = img->amgdata.bilspir: 
Linecount = 0: 


/* The scan of an RGB image is from the bottom row -> up, 
traversing the rows left to might. */ 


ford = Xdim + 131 < (AXdim*(img->ysize-1))-1; ++1) 


/* If pixcl iis in the lefunost column, do check_active_edges(). */ 
Wwa&% Adin == VY) check_acuuve_verucal_edgesQ; 


/* Ensure pixel iis notin leftmost or rightmost column */ 
eset dim $= Ndin-1) 


/* Calculate da,dy via Sobel operator for pixel 1. */ 


dy = (-(ptr[i+Xdim-!]&grmask) + (purtitX diim+1J& grmask ) 

-(2 * (pur[i- 1 |A&grmask))+ (2 * (purfit 1]& grmask)) 
-(pur[i- Xdim-1]&ygrmask) + (pur[i-Xdim+1}&grmask)); 

dy = ( (pur[i+Xdim-1]& grmask) + (2*(pur[i+Xdim]& grmask)) 
+ (ptr[it+ Xdim+]J&grmask) - (pur{i-Xdim-1]&grmask) 
- (2*(puli-Ndim}&grmask)) - (pur[i-Xdim+1]& grmask)); 


i ((dx* dn + dy *dy) > Th) pixel_membership(z,atan2(dy,dx)); 
se 


ie eendior i */ 


pep@iicck renaming EDGES for lines:*/ 
neg F aslecdee list head: 
merce = SULL} 


vertical_line_test(reg): 
reg = reg->ncats 


i amtenmeslines list to tile “lines.text . */ 
write_all_linesGimg->xsize,img->ysize); 

print{(® Number lines found in %s = %d\n” ,img->name,Linecount); 
return(Line_list_head): 
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APPENDIX C- LINE MATCHING AND POSE DETERMINATION ROUTINES 


A. IMPLEMENTATION: VERTMATCH.C 


[RR RE AEN EE Te A ea ee 


P PILE NAME enunaiciec 

/* AUTHOR: Kevin Peterson 

i> DA TEs 09 March 1992 

[RRR AAA AEE Ae EAS A A A AA ES A ee 
#include <gl.h> /* SiliconGraphics (1) graphic library */ 

#include <gl/image.h> /* SGI image structure library */ 

#include <device.h> /* Machine-dependent device library for keys and mousc-buttons */ 
#include <stdio.h> /* C standard 1/o library */ 

#include <math.h> /* C math library for atan2Q() */ 


#include “2d+.h” /*Header files for 3D model environment by J. Stein */ 
#include “Sth.h” 

#include ““graphics.h” 

#include “visibility.h” 


#include “image_types.h” /* Type definitions for NPSIMAGE, etc. */ 
#include “match_types.l /* Type definitions for EDGE, IMG_LINE.. */ 


#include “npsunagesupport.h” /* Some NPSIMAGE funcuons */ 

#include “cdgesupport.h” /* EDGE and IMG_LINE building funcuons */ 
#include “vertsupport.h /* Venical EDGE and IMG VINE sSupplcnicnn” 
#include “matchsupport.h” /* LINE and IMG_LINE matching routines */ 
#include “matchdisplay support.’ /* Graphics display funcuons */ 


main(arge, argv) 
Mts 
Char = ages 


WORLD Filth loch 

NPSIMAGE *inic: 

IMG_LINE *ImageLinesHead = NULL, *imageline; 

LINE. HEAD *ModelbinesHcad] = NUE = Modelomestcad = (ile 
POSE “those, ccimno.e 

float x0, yO, thetaQ, u_posil, center_u_axis; 


i (arge '= 2) fatalC*usawe: verumatch filename\n”); 
/* Pose values for data images: ------- 22-2 --e eee e eee eee eee a 


x0 = 60.0; y0 = 366.0; thetaU = 250.0: /* 060366250.pic */ 

Pe x0 = 60.073 0 = 366.0; them0= 25507" (603662540, 
/* x0 = 60.0; 5.0 = 366.0; inew0 = 231027 Dols GerS ies: 
/* x0'= 59.0: 0 = 366.0: incin0 = 250.0; 7" 059366249 pic, 
[* x0 = 94.0540 = 381-0: thet0 = 24530-7- 024351245 pices, 
/* xO'= 68.0: 50'= 372.05 the 249. 02" 06837224 ne 


/* x0=48.0; 50 = 277.0; ineu0 = 0000-7 462) ie 
/* x0 =48.0; yO = 277.0; theta0 = 002.0; /* 048277002.pic */ 
/* x0 = 480, ;0 = 277.0; thea0 = 3560 (36277 s3G pie 
/* x0 = 43.0; 50= 277.0; ihei0 = 35607 2 eee 
/* x0 = 43.0; yO = 267.0; thetaQ = 000.0; /* 043267000.pic */ 
/* x0 = 48.0; yO = 267.0; thetaO = 000.0; /* 048267000. pic */ 
[* x0 = 48.0; yO = 272.0; theta = 000 Gry a2 ee ic 
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/* Read input rgb image and extract verucal edges. */ 

img = read_sgi_rgbimage(argv[1]); 

ifamg ==(NPSIMAGE *) NULL) fatal(“File %s 1s a NULL image.\n” img->name); 

print{(“verunutch> %s asize= %d ysize= %d pixels= %d\n”, 
ing->namne,img->xsize,img->ysize, (IMg->xsize*1ng->ysizc)); 


ImageLinesHead = veruinesamg); 
imageline = ImageLinesHead; 
eentcr_U_axis = (loavjaing->asize/2): 
while(imagcline '= NULL) 


u_posit = (amageline->p1.x + imageline->p2.x)/2.0; 

imageline->angle_to_image_center = atan2((center_u_axis - u_posit), 
FOCAL EPNGHiH IN PIXELS); 

imuageline = imuageline->next; 


/* Declare the esumuated pose of the camera. */ 

iene pose (POSE wmalloc(sizcol Os e))) == NULL) 
HeteaMie Po oeanepose; malloc\n’):; 

IN_pOse->a = v0: 

in_pose->y = yO; 

in_pose->theta = normalhize(thetaQ * (PI/180.0)); 

Bone HOOK (input pesca, — 2, cl y—— 2[, theta = %.2!(%.2! rads)\n\n’, 
IN_POse->\,1n_pose->y,thetaQ,in_pose->theta); 


/* Imituahize world database for fifth floor of Spanagle Hall, determine 
2D view of environment, and label the vertical model features. */ 


PithFlour =make workl®: 
ModelLinesHead] = vet_view(in_pose->x, in_pose->y, CAMERA_HEIGHT, thetaQ, 
FifthFloor, FOCAL_LENGTH); 


aie anh \ cities odelieiies = “d\n, Modell inesHead |-> VER T_LINES): 
label_model_lnes(ModelLinesHead1->VLINE_LIST, in_pose): 

fC Calloupdatc poset) 7/ 

COr_poOse = update_pose(ImageLinesHead ,ModelLinesHcad 1-> VLINE_LIST,in_pose); 


thetaO = corr_pose->theta * 180.0 / PI; 
mete =< 0.0) theind += 360.0: 


prinu(¢\nCorrected pose: \= @.2f, y= %.2f, theta= %.21(%.4f rads)\n”, 
COM_pOse->x,Corr_pose->y,theta0,corr_pose->theta): 


/* Get 2D view of world from corrected pose. */ 
ModelLinesHead2 = yet_view(corr_pose->x, corr_pose->y, CAMERA_HEIGHT, theta0, 
Pine loon FOCALSEENGTH): 
/* Display image, 2D view trom input pose, and 2D view {from corrected pose. */ 
display_match_image(imyg. ImageLinesHead, ModelLinesHead1, ModelLinesHead2); 
free_lines(ModelLinesHead 1); 


free_lines(ModelLinesHead?); 


rintfC* vertmatch %s...done An” argv[1)): 
p g 
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FILE: MATCHSUPPORT.H 


dia Nai a stile a a Naa a ee IN NG BEI 


/* FILENAME: matchsupport.h 
/* AUTHOR: Kevin Peterson 
/* DATE: 09 March 1992 


[RRR EE RRR RR REE EE EAE EEE Ee ee ee ee 


#define FOCAL_LENGTH 1.40 /* cm */ 

#dcfine FOCAL_LENGTH_IN_PIXELS 1205.4524 /* number of pixels */ 
#define VERT_PICTURE_BORDER 8.0 /* number of pixels */ 

#dcfine FIELD_HALF_ANGLE 0.2556 /* radians = 14.645 degrees */ 
#dcfine CAMERA_HEIGHT 40.0 /* inches */ 


#dcfince MIN_CHORD_LENGTH 10.0 /* min dist between 2 vertical 
model features (in inches) 
used in determine_position()*/ 


#define DELTA_LENGTH_RATIO 0.05 /* Endpoint delta/IMG_LINE Iength 
used in determine_vert_match()*/ 


#define IMG_LINE_MIN ANGLE 0.0070 /* radians = 20idretces/ 
#dcfine VERIFY_EQ_ ANGLE EPSILON 0.0035)" radians — O20 ieerces 7 


ME A eh eR PURE = aly 2 sll nO) a ay 
/* double normalize qdoble alpha) 

(es 

/* Retums an angle between prand -pl. 

i* ae RR Fel ae Re OME MME oy 2G ars Tet (A AI A eee els i * / 


double normalize(alpha) 
double alpha: 


whileGaipha > Plpalphae=<(2.03Pi 
“hiteqalpha < -PI) alpha += (2.0* Pl); 
return alpha: 
je ee Sno aE Marte 3 ncn s Lad artes oh RE tie Pee ec ee Sac a ne 3 hp oe aa 
jie 
Returns Pil angle al 1s leitsol angie ac: 
Wes Cae eet tN ec OT Rk ST ae Dd ay, 
ag 


Int lelteolie 2) 
double lea. 


return (normialize(al - a2) > 0.0); 


ie w eS oc se et as eee ete ee ee eee sa! 


Returns angle for use with environment model coordinates. 
Environment model uses angles measured from y-axis. 


double map_angle(alpha) 
double alpha: 


return normalize(alpha - (PI/2.0)); /* alpha_prime */ 


}a2 


/* void label_model_lines(LINE *m, POSE *p) 

bia 

/* Routine to determine lengths, orientations, angles from image center, and 
/* labels for all model LINEs visible in image. 


label_model_lines(m, y)) 

EINES ine 

POse ay  stinaied camer OSE */ 
{ 


insiincd chr tirstechw = second chur= A’; 


/* Note: Verucal edge extracuon produces IMG_LINEs with pl.y always 
less than p2.y. Similarly for vertical model LINEs, Y1 is always 
less than Y .. 


4 
printiC\nModel Lines: ------ wie 


nein t= NULL) 


{ 


m->length = m->Y2 -m->Y1; 

M->est_poOse_orient = map_anglie(atan2(m->MODEL_Y - p->y, 
m->MODEL_X - p->x)); 

m->est_angle_lo_image_center = m->cst_posc_oricnt - p->thela; 


iun->length > 15.0) /* temporary -- don’t do very small lines */ 
sucepy(m->name,” “): 
sprinuini->name, "%c%c", first_char, second_char); 


Wisccond_char == °Z’) 
| 
MCirstechar ame IFSE Chan = A - 
Ce itstechair = iirstecnar-+ |: 
Seeanduchar = A.: 
| 
J 
Cleesccond chur = sccond chun: 


print” %s: (%.16,%.1f) length= %.2!, psi= %.2f, psi-theta0= %.4{\n”, 
m->name, m->MODEL_X, m->MODEL_Y, m->length, 
m->cSt_posc_oricnt, m->est_angle_lo_image_center); 


sprintl(m->name, **); 


Ly Sse NIE IE 


Ales: 


/* void determine_vert_match CINE “meal MGeeiNiE sn 

ris 

/* Function to determine confidence value for a possible match 
/* based upon translauon, rotation, scaling, and overlapping 

/* of endpoints between a LINE from the model list and an 

/* IMG_LINE from the image I1st. 


determine_vert_match(m,1) 
LINE *m; /* pu to LINE from model list */ 
IMG_LINE *1; /* pu to IMG_LINE from image list */ 


int endpoint_inclusion; 
double delta; 
MATCH) PE “inatehe hice 


/* Endpoint_Inclusion of image endpoints: for vertical image lines, 1->p1 is 

always lower than 1->p2. */ 

delta = (m->Y1 - 1->pt.y) + U->p2.y - m->Y2); 

f((delta <= 0.0) I (delta > 0.0) && (delta/i->dmajor < DELTA_LENGTH_RATIO))) 
| /* then create NIAT CHT ye indininccan the TMI aIe Nie sainatelnis ie, 


if(anatch = (MATCH Y PE * nitallocsizeci( VAT Clit 4ee)) == NUTS) 
fatal(“derermine_match: creating MATCHTYPE, malloc\n”); 


niaich-siine =n, 
match->angle_view_diff = (m->est_angle_to_image_center - 1->angle_to_image_center); 
INgdich-Snexl= NUE: 


Place match in imatchlist by increasing order of angle_view_diff, */ 


it (sraichhst == (NUE ine mite nist = smiatein 
else W(fabs(miatch->angle_view_diff) < fabsG->matchlist->angle_view_diff)) 


mutch->next = i->matchlist 
i Sieichlist jm aren 


else 
] 
t 
place = 1->matchlist; 
while((place->next != NULL) && (fabs(match->angle_view_diff) >_ | 
fabs(place->next->angle_view_diff))) 
J 
l 
place = place->next; 
match->next = place->next; 
place->next = match; 


ya Cn ienae 7 
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/* void find_vertical_matches(IMG_LINE *imgline, LINE *modelline) 
ie 
/* Function to find all possible vertical model LINEs that could be matched 
/* to a vertieal IMG_LINE. 
(bs mts pt RE a 5 | ae NUE ter ct, SSG OY ala SME ee np io ee | 
find_vertieal_matehes(imglinec,modelline) 
PIGLEINE Amelie: 
Pie “modelline: 
{ 


while (modelline '= NULL) 
determine_vert_mateh(modelline,imgline); 
modelline = modelline->NEXT; 


ingline->pm = imeline->matchlist; 


— ae EN Ps a a a 
void printamaichlisiihiG_LINE *)} 

je 

/* Function to output the model LINEs matched to an IMG_LINE. 
Ie ps cp ES ae ek SE */ 


print_matchlist(]) 
PGi EINE *!: 
{ 


Pieler ih = [->matchist 
i a ae ee 


print” “¢s(%.4f)", m->line->name, m->angle_view _diff); 
Pie. ee ee 


print{(\n\n"): 


} 
ie yee ste coe ee ce i a Se ey Sl RU ge NN a el gt SAR ESA A * 
Fo vOomldcciure Iiatches(iNiGs INE 71, 1NiG_ LINE *m, IMG_LINE *r, 
i LINE *modellines) 
* 


/* Declare all matches to model LINEs for left, middle, and right 
Pevertical | NG. LINES. 


declare_matches(,m,r,modellines) 

DG IetNE le sine 

INE *imnodcilines: 
{ 
find_verucal_matches(1,modellines); 
find_vertieal_matehes(in,modellines); 
find_vertieal_imatches(r,modellines); 
prinui(\nImage line matchlists to Model line NAME(angle difference): ----- A: 
prinuf(‘left (edge %s) > \n”,]->name); print_matehlist(1); 
print{(““middle (edge %s) > \n”,m->name); print_matehlist(m): 
printf(‘right (edge %s) > \n”,r->name); print_matehlist(r); 
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lies oo eras es wis Sia cob ws SSS is Se BS “af 
/* int determine_posuont(LINE *1, LINE *m, LINE *r, double alphal, double alpha2, 
le POSE “prdouble 2 double, 
[= 
/* Rouune to determine correct position based upon 3 matched model LINEs and 
/* the measured angle differences from the IMG_LINEs. Retums 1 if x,y 
/* posiuon ts determined, 0 otherwise. 
2k 


int determine_posiuon(|, m, r, alphal, alpha2, p, x, y) 
LINE *1, *m, *r; /* left, middle, and right viewed mode! LINEs */ 
double alphal, alpha2; /* measured image angles */ 
POSE *p, /* estimated inp!) ROSE, 
double *\, *y; /* x.y location (6 be merinned 7/7 


{ 
double chord1]_length, chord2_length, 
chord]_mid_x, chord1_mid_y, chord2_mid_x, chord2_mid_y, 
chord1]_orient, chord2_orient, 
hi Oficnl no orients a ene: 
DV lok pVie DV OexX Ney. 
Circle eriGdiis circles wradiice 
Circle Fox, Circle] cy semele Zax ciclezave 
cc], cle llonent; «2 1rebotonlent cz. comers 


/* Ensure that 3 ditlenent modeiiNes arc bewmmuised a.” 
fl ==) it Git =a le — ie) 


prnuCoNO position: 2 1MG_LINEs matched to same model LINE\n”); 
return Q: 


/* Determine chord lengths, midpomts, and orientauons. */ 
chord!_leneth = 
sqrl(((l->MODEL_X - m->MODEL_X )*(I->MODEL_X - m->MODEL_X )) + 
(J->MODEL_Y -m->MUWEL_Y )*(->MODEL_Y - m->MODEL_Y ))); 


chord2 tens t= 
sqri((din->MODEL_X - r->MODEL_X )*(m->MODEL_X - r->MODEL_X )) + 
(un->MODEL_Y - r->MODEL_Y )*(m->MODEL_Y - r->MODEL_Y ))); 


if(chord!_length < MIN_CHORD_LENGTH) Il (chord2_leneth < MIN_CHORD_LENGTH)) 
{ 

prinui(*NO position: chord length < MIN_CHORD_LENGTH\n”); 

return QO: 


chord] mid_a = (->MODEL_X + m->MODEL_X )/2.0; 
chord]_mid_y = (l->MODEL_Y + m->MODEL_Y )/2.0; 
chord2_mid_x = (n->MODEL_X + r->MODEL_X )/2.0; 
chord2_mid_y = (m->MODEL_Y + r->MODEL_Y )/2.0; 


chord? _orient = Map_angle(atan2(l->MODEL_Y - m->MODEL_Y, 
l->MODEL_X - m->MODEL_X)); 

chonl2_oriem = map_angleGitan2Gn==MODEL_Y =). 1 ee ie 
m->MODEL_X - r->MODEL_2)); 


/* Determine orientations perpendicular to the circle chords towards input pose. */ 


hl_orient = normalize(chord]_orient + (P1/2.0)); 
h2_orient = normalize(chord2?_orient + (P1/2.0)); 
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/* Determine perpendicular lengths from the center of the chords to a point on the viewing circle. */ 
hl = (chord]_length/2.0) / tan(alpha1/2.0); 
h2 = (chord2_length/2.0) / tan(alpha2/2.0); 


/* Determine the viewing points perpendicular to the chords. */ 
pvl_x = chord]_mid_x - (hl * sin(hl_orient)); 

pvl_y = chord]_mid_y + (hl * cos(hl_orient)); 

By 20% —chora2_inid_x=(h2~ sin(i2_orient)); 

pv2_y =chord2_mid_y + (h2 * cos(h2_orient)); 


/* Determine radii and centers of viewing circles. */ 
circle]_radius = sqrt((h1*h1) + (chord1_length*chord1_length/4.0))/2.0; 
circle2_radius = sqri((h2*h2) + (chord2_length*chord2_length/4.0))/2.0; 


eicle lex = pv lex - (circle) radius * sin(h!_orieat + PI)); 
emcee ly — pv ly + (eirclel radius * cos(h!_orient + Pl)); 
emcle2_ xX = pv 2.x -(cimle2 radius * sin(h2_orient + PI)); 
emcle2_y = pv2_y + (circle2_radius * cos(h2_orient + PI)); 


/* Determine distance and orientation between centers of viewing circles. */ 
ele sun (ciicleley - circle? x)* (circle1_x = circle2_x)) + 
Ncieioley —cimcles., ) (encleley = circle? -y))): 


tee Cl == (),0) 

( 
printi(°NO position: viewpoint circles identucal\n’’): 
return 0: 


ee olmonent — tap ancien (circle! - circle2_y, circle!_x - circle2_x)); 
c2robol_onient = map_angle(atan2(p->y - circle2_y, p->x - circle2_x)); 


/* Calculate viewing position. */ 


mc rmele2 midis < labs(c2c] = eirclel_radius)) c2_omega = 0.0; 
me oicmele radius =(¢C2cl-+ circle! ridius)) ¢2_ omega = Pl; 


eee momen — iconic 7cl “c2cl) + (circle? nadius* circle? radius) - 
foie leimradius ~ cCimeleleradms) jy (2 * ¢2cl * circle2 radius)): 


metteol(c2robol_ orient, ¢2c1l_orient)) 


e— Cielo (circles radius ~ Sin(eZcl_onem+c2 omegi)); 
Sy =circle2 y= (circle? radius * cos(c2c)_oricnt+c2_omepa)): 


Else 
“x = circle2_y - (curcle2_radius * sin(c2cl_orient-c2_omega)); 
= cirele mes nerrcle2 radius * COs(c2cll_Orient-c?2_omega)); 
} 
FoUurn |: 
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% 7 


/* void sclect_neat_best_match (MATCH YPE "sii Gr Pies ine 
/* MATCHTYPE *r, int *k) 
am 


/* Sclect next best model LINE (minimum fabs(angle_view_diff)) for new 
/* combination in the neat determine_position() atlemplt. 


sclect_next_best_match(],m,r,k) 
MATCHTYPE *1, *m, *r; /* MATCHTYPE pointers for left, middle and right 
IMG OLINES =} 
WN oe /* number of possible pm->next != NULL */ 


{ 
Swiicnte hk! 
{ 
CSC 5: 
if((fabs(->next->angle_view_diff) < fabs(m->nexl->angle_view_diff)) 
&& (fabs(l->next->angle_view_diff) < fabs(r->neat->angle_view_diff))) 
* keels 
else if(labs(n->next->angle_view_diff) < fabs(l->next->angle_view_diff)) 
&& (fabsim->next->angle_view_diff) < fabs(r->next->angle_view_diff))) 


eS 
fed Foret Cae Gs 
DEG IRG 
Cisia 
i.U-Snest == N VEE) 
i(fabs(m->neat->angle_view _diff) < fabs(r->next->angle_view_diff)) *k = 2; 
else *k = 3; 
Clse 1 (nlesne xt == Nee 
f 
| iffabsql->neat->angle_view _diff) < fabs(r->neat->angle_view_diff)) *k = 1; 
ah<Co =. 
] 
cise 
H(Uabs(->next->angle_view_dilf) < fabs(n->next->angle_view_diff)) *k = 1; 
Cis “he 
| 
} 
PGC he 
GSC. 
MdeSnext = Ue) lee al 
cles = NUE 
clsek =a 
break: 
default: 
a Oe 
Dre ik. 


17 cndswaren 7; 
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/* double detcrmine_rotation (LINE *1, LINE *m, LINE *r, double x, double y, 

a double intheta, double loffang, double moffang, 
ie double roffang) 

* 


/* Function that determines the average amount of rotational correcuon 
/* that is required based upon the 3 model LINEs angular offset from the 
/* IMG_LINEs to which they are matched with. 
eg eI ene et Sn Teen ET NI ee ae | 
double determine_rotauon(l, m, r, x, y, intheta, loffang, moffangy, roffang) 
INE ee ie or 
double x, y, intheta, loffang, moffang, roffang; 


/* calculate orientauions of new position to Vertical Model LINEs */ 

double Ineworient = map_angle(atan2(l->MODEL_Y - y, l->MODEL_X - x)), 
. y P 

mneworient = map_angle(atan2(m->MODEL_Y - y, m->MODEL_X - x)), 

rmeworient = map_angle(atan2(r->MODEL_Y - y, r->MODEL_X - x)), 

/* calculate angles from center of rmage for new position */ 

Inewestangle = normalize(Ineworient - intheta), | 

mnewestangle = normalize(mneworient - inthcta), 

mewestangle = normalize(meworient - intheta), 

/* calculate differences with offset angles from image */ 

|_diff = normalize(Inewestangle - loffang), 

m_diff = normalize(mnewestangle - moffang), 

r_diff = normalizecmewestangle - roffang), 

/* average rolavion diflerence */ 


rotauion_dilf = (I_diff + m_diff + r_diff) / 3.0; 


return rotauvion_ dill: 
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mem me ee Pee Ee ee EE EE Pe Ee ee Ee Ee ee EE eee ee ee ee ee 


/* int verify_pose(double x, double y, double theta, IMG_LINE *ImgLuines, 
ee LINE *ModelLines) 

ei 

/* Funcuion to return number of IMG_LINEs that lie over model LINEs 

/* for a corrected 2D view of the model environment for a given possible pose. 


int verify_pose(x,y,theta,lmgLines,ModelLines) 
double x,y,theta; 
IMG_LINE *ImgLines; 
LINE, *Modelleires: 


{ 

EINE *ml = Modellainegs: 

IMG SEINE 71] =hngkines, 

double delta; 

int endpoint_inclusion = Q, imgline_on_modelline = 0, nr_good_hits = 0, 
found = 0; 


/* Calculate est_angle_lo_image_center for all ModelLines based upon 
pose: (x. y,lheta). */ 
whiednlt= Cie 


{ 
ml->est_angle_to_image_center = 
normalize(map_angle(atan2(ml-> MODEL_Y-y, ml->MODEL_X-x)) - theta); 
=n NE. 
} 


/* Find correspondences of image lines to model lines. */ 
Woche = IN ee 


{ 
/* Ensure verucal image line is not due to image borders. */ 
f(absdl->angle_to_image_center) < FIELD_HALF_ANGLE) 
/* Find a corresponding IMG _LINE® «7 
ml = ModelLines; 
ound 0: 
whileqinl '=NULE) &c ound) 
della = (ml->Y1 - il->pl].y) + (il->p2.y - ml->Y2); 
endpoint_inclusion = 
((delta <= 0.0) Il 
((delta > 0.0) && (delta/il->dmajor < DELTA_LENGTH_RATIO))); 
imgline_on_modelline = 
(fabsCil->angle_to_image_center - 
ml->est_angle_lo_image_center)<VERIFY_EQ ANGLE_EPSILON); 
if(endpoint_inclusion && imgline_on_modelline) 
++nr_good_hits;: 
found =a) 
n= mls NENT: 
~ 
a jee 3 ea 
return nr_good_hits; 
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peeose ellcumicencsteposciiMG LINE *l, IMG_LINE *m,IMG_LINE *r, 


/* doubleaiphiadlinedeuble alphalmr, POSE *in pose, 
Vhs DIGeEINEimelines, LINE “Modell ines) 
Veg 


/* Function that conducts several iterations of finding an IMG_LINE to LINE 

/* triplet, determines the POSE based upon the matching, and returns the best POSE 
/* based upon number of IMG_LINE to model LINE matches of a 2D view of the 
/* best POSE and translational error of the robot. 


POSE *calculate_best_pose(1,m,r,alpha_lm,alpha_mr,m_pose,ImgLines,ModelLines) 
IMGOEEINE im, F, imeLines: 
double alpha_Im, alpha_mr; 
POSE ne nosc: 
EINE Niodel@ines: 


{ 
int a re ih nits — best mi its =U: 
EOSE | besSis pose: 
double x, y, Fol, wans = 0.0, best_rot, best_trans: 
heer rly PE inaich = i-emaicniist, 

*mmatch = m->matchlist, 

Aamaich = F-Smatchlist: 
printi(’Pose determination: ----- Aled: 


feist pose = (POSE * malloc(sizcol(POSE)))} == NULL) 
fatal’ caleulate_best_pose: malloc\n”); 


Meni a a eeletinnriel == INCE) itrmatch == NULL)) 
J 
printh “Can not determine pose for %s, %s, %s (NULL maitchlist)\n”, 
!->name n->name,r->mame): 


J 


elector = 0; 1< 20; +44) 


printhe” [Scs,%s.Scs} “Imatch->line->name,mmatch->line->name, 
rmatch->line->name): 


(determing  posiuon(imatch->line, mmatch->line, rmatch->line, 
alpha_Im, alpha_mr, in_posc, &x, &y)) 


/* Calculate translation and rotauon differences. */ 


rans = sqri((in_pose->x - x) * (in_pose->x - X)) + 
((in_pose->y - y) * (in_pose->y - y)); 


rot = determine_rotation(Imatch->line, mmatch->line, rmatch->line, 
X, y, IN_pose->theta, 
l->ungle_to_image_center, 
m->angle_to_image_center, 
r->angle_to_image_center); 


nr_hits = verify_pose(x,y,m_pose->theta + rot, lmgLines,ModelLines); 


prinui(’x= %.2f y= %.2f T= %.4f R= %.4{ nr verify matches= %dw , 
x,y,Uans,rot,nr_hits); 


if(Q == Q) Il (((double)nr_hits)/trans > ((double)best_nr_hits)/best_trans)) 
{ 


best_nr_hits = nr_hits; /* save best trans and rot values */ 
best_trans = lrans; 

DESLTel — Te 

best_pose->x = x; /* assign values to best_posc */ 
best_pose->y = y; 

best_pose->theta = in_pose->theta + rot; 

print’ *** BEST POSE MODIFIED “22 1). 

1->pm = Imatch; /* set 3 IMG_LINE’s match pointers */ 
m->pm = mmatch; 

r->pm = rmatch; 


} 
/* Select the next best match combination of model LINEs. */ 


k = (imatch-Ssnext '= NULE) Biinitch- nex = NUE 
(rintatch-sheat'= NUL 


ihe 
select_next_best_match(Imatch,mmatch,rmatch,&k); 
ith == 1) latch =e ne xt. 
elscil(k == 2) mmatch = mimateh=snext: 
cls i(k == 3) nmaich = fimelcn- new: 


Clse return best_pose: 


return best_pose: 


eos Updalemmosct NG eelNeimareimnes, LINE =Modellines, POSE *in_pose) 
[= 

/* Function to return a corrected POSE given the vertical edges extracted from an image 
/* and the vertical model LINEs that should be visible from the expected POSE of the 
/* robot. 


POSE *update_pose(ImageLines,ModelLines,in_pose) 
IMG_LINE *]mageLincs; 
LINE *ModelLines; 
BOSE innmase: 


IMG_LINE *il, *Ieft, *middle, *right; 


BOSE *QUL_pose; 
double alpha_Im, alpha_mr; 
int J: 


Moi pose = (POSE “i mnalloc sizeol(POSE))) == NULL) 
fatul(update_pose: creating POSE out_pose: malloc\n”): 


/* Find the 3 most significant IMG_LINEs and return them in 
left-to-right order pointed to by *lefl, *middle, and *right. */ 


il = ImageLines: 
printh(N\nDetermine 3 mayor image edges for matching: ----- \n’’); 


forG = 1; ) < 4:) 
i ele i Grown ) DETERMINE POSE: < 3 image Jincs\n’ ): 
/™ Ensure verucal image line is not due 10 image borders. */ 


Nae ine loninace. center) < FIELD HALF ANGLE) 


} 


Lb 
leu 1): 
prinul(” %d) line %s, angle from center = %.4f{\n”, J, 
il->name, il->angle_to_image_center); 
sata 


else tg == 2) && (labs(il->angle_1o_image_center - 
left->angle_to_image_center) > IMG_LINE_MIN_ANGLE)) 


if(il->angle_to_image_center > lelt->ungle_to_image_center) 


else mahit = ilk 


right = left; lefi = il; 


prinu(* %d) line %s, angle from center = %.4{\n”, ), 
il->nume, 1l->angle_lo_image_center); 


++): 


else 1h ==3 eee 
(fabs(l->angle_to_image_center - 
Ieft->angle_to_image_center) > IMG_LINE_MIN_ANGLE) && 
(fabs(Gl->angle_to_image_center - 
right->angle_to_image_centcr) > IMG_LINE_MIN_ANGLE)) 
i{(il->angle_to_image_ccnter > Icft->angle_to_image_center) 


middie —letevett—ailk 


else if(il->angle_to_image_centcr < right->angle_to_image_center) 


middle = right; right = il; 


} 
else media = alk 


printi(” %d) line %%s, angle trom center — 70.41 aa), 
il->name, 1l->angle_to_image_center); 


ase 


1) = il->nexk 


/* Calculate two angles between the three major IMG_LINEs. */ 

alpha_Im = left->anyle_to_image_center - middle->angle_to_image_center; 
alpha_mr = middle->angle_to_image_center - nght->angle_to_image_centier; 
/* Declare the matches 10 nredel LINEs for the Ihree major |G lines) 


declare_matches(left. middle ight, ModelLines); 


f Caleulare ihe Comcoliosc 3, 


Out_pose = culculate_best_pose(left, middle, nght, alpha_lm, alpha_mr, 
in_pose, ImageLines, ModclLines); 


rclurn OUL_POse: 
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FILE: MATCHDISPLAYSUPPORT.H 


RR AE RE REAR EHR ERR IR KK KOR / 


/* FILENAME: matchdisplaysupport.h 
iP sO THOR; Kevin Peterson 
PeDATE 07 February 1992 

* 


/*DESCRIPTION: Collecuion of display functions. 
Hh 


/* void display_match_image (NPSIMAGE *img, IMG_LINE *I, 

ig EINE SHEAD “ml EINE HEAD *m2) 

/* void draw_white_lines(IMG_LINE *1) 

/* void draw_white_model_lines_with_names (LINE *m) 

/* void draw_whitc_model_lines (LINE *m) 

/* void draw_black_modcl_lines_with_names (LINE *m) 

fF void draw_black_model_lines (LINE *m) 

f* void draw_match_lines UMG_LINE *I) 

/ void display_match_loop (NPSIMAGE *img, IMG_LINE *1, 

ie PiviieheaD mi. LINESHEAD “m2, lone winid) 


/* These functions make calls to following SiliconGraphics routines: 
/* prcfsizeQ), winopen(), winset(d), winclose(), RGBmode(), 

/* singlcbuficr(y), gconlig(), ydeviced, c3f0, move2(), draw2(), 

/* swapbuffers(), Irectwrite(), reshapevicw port(), cmov2(), 

/ charsir(), and clcar(). 


ee Ag ee a EES Ee EE EK REE EE ES AM / 


/* void display_match_image (NPSIMAGE *img, IMG_LINE *1, 
P2eiNeene Aw *n), LINE HEAD *m2) 


display_match_image(mg,],m1,m2) 
NPSIMAGE *img; /* input image */ 
Poel l-7- listol IMGSEINES found lironi image */ 
Need ean), m2) iNeereAD types irom Jim s eraphics.c” */ 


long winid; 


prefsizcimg->asize,imy->ysizc); /* preferred size for window */ 
winid = winopen(img->name); /* open the window */ 
RGBmodc(); /* set RGBmode, singlebuffer, and */ 
singlebuffer(); /* configure the window */ 

gconfig(); 


/* intualize_controls */ 
qdevicc(REDRAW),; 
adevice( LEPT MOUSE): 
qdcevicc(MIDDLEMOUSE): 
qdevice(RIGHTMOUSE): 
qdevice(ESCKEY): 


display_match_loop(img,],m1,m2,winid); 
freeaamg->imgdata.bitspu); /* dclete the bitmap for the image */ 


free(img); /* deletc the NPSIMAGE structure */ 
winclose(winid): /* close the window */ 


a void draw _white_linesaMG_LINE *]) 
Draws white lines over an image. 
draw_white_lines(1) 
[ING aEINE | 
eee float white[3] = {1 O}R O30) 77 tebe ite 


c3f(white); 

wed) = tNig lel 
cmov2(]->p1.x,l->pl.y); charsur(l->name); 
move2(1->p1.x,l->pl.y); draw2(1->p2.x,]->p2.y); 
[TS ext 


swapbulfers(): 


/* void draw _whilte_model_ lines: withinaimes (ei ee 
/* Draws blue model lines over an image. 


draw_white_model_lmes_with_names(m) 
LN 


stalic float whites |= (1 OOO) 7 re Bite 


C3i(w Hite): 
Wihiletny = Lasley 
{ 


cmov2un->X 1,m->Y1); charstrun->name); 
mMoVvVel (MES N | ae yada i ie 
i= Nes eee 


swapbulffers(): 
edema ns cpm ee Wcities =, 
/* void draw_black_niodel_ lines with names (cE INE “im 


/* Draw’s black model lines over an image. 


[7 one cere nc ce cwncrn nnn ew ener ern en renee cece ners tence ener nmmecntenes a 
draw _black model_lines_with_names(m) 
EINE in: 
{ 
staue float blick|3)= (0.0,0.0,010} 7 rep blacks =) 
c3f(black); 
Which! =U) 
{ 


cmov2(m->X 1,m->Y 1); charsur(m->name); 
move2(m->X 1,m->Y1); draw2(m->X2,m->Y 2); 
ie ese 

} 


swapbulfers(): 


/* void draw _white_model_lines (LINE *m) 
* 


/* Draws white model lines over an image. 


draw_white_modcel_lines(in) 
LINES 


{ 
Sac Mout wine = IOI) 10): 7 reb white */ 


c3f(white); 
whileGni=NULL) 
{ 


move2(n->X 1,m-> Y1); draw2(m->X2,m->Y2); 
i Sil Neen P: 


swapbullers(): 


/* void draw _black miodel_lines (LINE *m) 
ei 
/* Draws black model lines over an image. 
ie te ee a A ar eae a 
draw _ black model linestn 
lees tie 


Sie tone black (3s) = 404) 0,00): /* reb black */ 


eoliblack): 
Cul: =n LL} 
{ 


move2(m->A ],mi->Y1): draw2(m->X2,m-> Y2): 
Nie Ne 
swapbullers(: 


i <r es ght wc eee * 
/* void display_match_loop (NPSIMAGE *img, IMG_LINE *1, 
/* LINE_HEAD *m1, LINE_HEAD *m2, long winid) 


display_match_loop(img,l,m1,m2,winid) 
NPSIMAGE *img; /* input image */ 
IMG_LINE *1; /* list of IMG_LINEs found from image */ 
LINE HEAD *m1,*m2; /* LINESHEAD ty pc trom Jimi Soran lea, 
long wind, 
static float white|3) = {1 OF ONO). Tepe intes 
short valuc; /* value returned from the event qucue */ 


/* display the images once */ 


winsel(winid); 
lrectwritc(O,0,1mg->xsize - 1,img->ysize - 1,img->imgdata. bitspt); 
draw_whitc_lines(l); 


/* loop until a mouse bullon is pressed */ 
while(TRUE) 


switch(gread(& value)) 


caso REDRAW - 
winscl((long)valuc); 
reshape view porlt(); 
We == ard) 


Irecuw rite(0,0,img->asize - I,img->ysize - 1, img->Imgdata.bilsptr); 
draw _whuite_lines(]); 


DOC 


Case LEPT MOUSE: 
ionic == 0) 
Irectw rite(0,0,1mg->asize - 1,img->ysize - 1, img->imgdala.bilsptr); 
draw _white_model_lines_with_names(m1->VLINE_LIST); 
draw _while_ model linesdiml=2INESEIS i: 


Drecik: 


case MIDDLEMOUSE: 
ivalic==70) 


{ 
Colle: 
clear: 
draw_black_model_lincs_ with_names(m1->VLINE_LIST); 
draw _black_model_lines(m1->LINE_LIST); 
Drceh: 


CASGRIGH TMOUSE: 
i Sali =—al)) 
{ 


Irectwrite(O,0,img->xsize - 1,img->ysize - 1, img->imgdala.bitsptr); 
draw _white_model_lines(m2->VLINE_LIST); 
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draw _white_model_lines(m2->LINE_LIST); 
} 
break; 


Case ESCKE\- 
return: 


default: 
break: 


} /* end switch */ 


} /* end while */ 


APPENDIX D- USER’S GUIDE 


file: README 
date: 8 March $2 
by : Kevin Peterson 


USER'S MANUAL FOR YAMABICO’S VISIUAL NAVIGATION @ReOUT INES 


This file outlines the functions cr the ftolvouwinge crear am.. 
findedge, fastedge, and vertmatch 


[2] REOUlREME as. 


1.1 HARDWARE: The above routines were developed on a 
SiliconGraphics Personal Iris workstation and can be run on 
all SGE statics 22eieo- 


1.2 SOFTWARE: Ensure the fcllowing files are in the 
local directory for the edge extraction routines: 
READ Mies ti iS ee) 

Makefile 

IMeagGereype sa 

Match typesak 

npsimagesuppeort.n 

edgesupport.ft. 
Vert Suppo me 
displaysupes:r 
findedge. ie 
fastedge.c 


bc - 
Sa die 


Ensure tne fs.lsenwing files are in the lecal adirecten, tense 
matching Diss SSterminat2on routines: 

Snel Teas 

2a 2h 

Grerr. ces 


deems 1 
Mawehsi@e ie. 
VERtmatcii. 


2 LCOMET Ga ies. 


2.1 The fo_icwlino C program libraries are required: 


oA 
gl/image.r. 


2.2 Lindeige, sea 
and vertmatch car. 
cc =o findedge = 
cc ~-o fastedge f 
Seas 


sasedge, fastedge, fastsobel, vertedge, 

all be compiled by the command line similar to: 
indedge.c -lamage =i sein 
astedge ¢ =lLimeges= lol issa 


or if the Makeriie is present: 
make findedge 

make fasteage 

Se caren 
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3. PROGRAM FUNCTIONS: 


3.4 FINDEDGE - Displays the grayscale and gradient 
images of the input image. Conducts edge extraction based 
upon least squares fit. 


3.6 FASTEDGE - Displays the input image and edges 
extracted. Uses the green component of the RGB pixel value 
vice the grayscale value and does not create the grayscale or 
gradient images. 


Peo SRIME@C Heese racts vertical edges from the input 
image (VERTEDGE), creates a 2D view of the model environment 
based upon the estimated pose (dead-reckoning maintained 
Pencigurationj ot the robot, conducts line matching between 
image vertical edges and model vertical line segments, 
determines possible poses based upon different matchings for 
the three major image edges, then returns the best pose based 
upon number edge to model line matches and distance of 
updated pcse tc the estimated (input) pose. 


Cal tats 


4.4 findedge <filerame> 


Potem@eateaw ds 112. cacture file to extract edges from. Two 
windows wil. be created; one for the grayscale image, the 
other tor the gradient image. 

- Lines extracted can be drawn onto either image by the 


Mer teee MIDOLEwmouse buttons while the cursor is 
Meee Oc 2 eee 
Sette sSeca f= Gran. over a white background by MIDDLE 


Gersemreotlciewn22e che cursor iS in the WINDOW. 
See te stots 8) _Eet miouSe burtons (cursor in WINDOW) or escape key. 


4.6 fastedge <riiename> 
One window is cpenéd to display the input (color) image with 
extracted edges arawn over the image. 
- Lines aes oe can be drawn onto either image by the 

Perl Meuss Pe@eto.s while the eursor is in the WINDOW. 
pete S Se - Cre.e Cver a white background by MIDDLE 

Mouse SOCL oh while tne cursor 1s in the WINDOW. 
eee: Gert meese CULTONS (cursor an WINDOW) or escape key. 


pe 


ry Psy 


4.9 vertmatch <filename> 

One window is cpened to display the input image with 

the extracted vertical edges. 

- MIDDLE mcuse button (while in WINDOW) displays the 2D wire-frame 

view for the plea pose over a white background. 

SR Gh «10 o Uttor displays 2D expected pose wire-frame view over 
the 

BOUL pleture., 

- LEFT mouse button displays 2D wire-frame view from the corrected 
pose 

over the in 

= Escape ke 


Cot 
Vo 2: 


! 


[BAL82] 
[BEV90] 


[FEN90] 


[HALS89} 


[HAR89] 


HEI) 


[MAR79 | 


[MOFS80] 


[KANSYa] 


[KANS89b] 


[KUM89]| 
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